Article image extraction method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides an article portrait extraction method, device, electronic device and storage medium, the method comprising: extracting the subject probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determining the subject keyword set of each article based on the subject probability distribution of each article; clustering all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; performing keyword extraction on each cluster in the multiple clusters to obtain the cluster keyword set of each cluster, and determining the cluster center article title of each cluster; for each article, determining the article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs; determining the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs, thereby improving the accuracy of subject keyword extraction, enhancing the generalization of recalled articles, and improving the recall quality of the recommendation system.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method, device, electronic device and storage medium for extracting article images. Background Art

[0002] With the continuous development of computer technology, more and more users obtain content of their interest through the Internet, and the recommendation system generates a recommendation list of users' interests based on the users' historical records or interests and hobbies and recommends it to the users.

[0003] In the related art, the Latent Dirichlet Allocation (LDA) model is applied to extract article portraits in the recommendation system, and the topic and topic keywords of the article are extracted through the LDA model. Specifically, the articles in the corpus are passed through the LDA model to obtain multiple topics corresponding to all articles in the corpus and the topic probability distribution of each article, and each topic includes multiple topic keywords; however, the multiple topic keywords in the multiple topics of each article overlap with each other, or contain some noise words, resulting in inaccurate extraction of topic keywords, affecting the recall quality of the recommendation system. Summary of the invention

[0004] In response to the problems in the related art, an embodiment of the present invention provides an article portrait extraction method, device, electronic device and storage medium, which are used to solve the problem of inaccurate extraction of subject keywords.

[0005] Specifically, the embodiment of the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a method for extracting an article portrait, the method comprising:

[0007] Extracting the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determining the topic keyword set of each article based on the topic probability distribution of each article;

[0008] Clustering all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; extracting keywords from each of the multiple clusters to obtain a cluster keyword set for each cluster, and determining the cluster center article title of each cluster;

[0009] For each article, determining article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs;

[0010] Based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs, an article portrait of the article is determined.

[0011] Furthermore, determining the subject keyword set of each article based on the subject probability distribution of each article includes:

[0012] Based on the topic probability distribution of each article, select the N largest topic probabilities in the topic probability distribution of each article; N is an integer;

[0013] Based on the topic keywords corresponding to the selected N maximum topic probabilities, a topic keyword set for each article is determined.

[0014] Furthermore, extracting keywords from each of the multiple clusters to obtain a cluster keyword set for each cluster includes:

[0015] Based on the multiple clusters, the number of times a keyword of each article in each cluster appears in the article is counted; the keywords with the largest number of appearances in each article are combined to obtain a cluster keyword set of each cluster.

[0016] Furthermore, determining the cluster center article title of each cluster includes:

[0017] Based on the multiple clusters, respectively calculating the distance between the vector corresponding to each article in each cluster and the vector corresponding to all articles in each cluster, and determining the cluster center vector of each cluster;

[0018] Based on the cluster center vector, the cluster center article title of each cluster is determined.

[0019] Furthermore, determining the cluster center article title of each cluster based on the cluster center vector includes:

[0020] Based on the cluster center vector, determine in each cluster that the article corresponding to the vector closest to the cluster center vector is the cluster center article;

[0021] Based on the titles of the cluster center articles, the cluster center article titles of each cluster are determined.

[0022] Furthermore, for each article, determining the article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs includes:

[0023] Based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, determining the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs;

[0024] The article keywords of the article are determined based on the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, and a preset number of cluster keywords in the cluster keyword set except the intersection.

[0025] In a second aspect, an embodiment of the present invention further provides an article image extraction device, the device comprising:

[0026] An extraction module, used to extract the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determine the topic keyword set of each article based on the topic probability distribution of each article;

[0027] A clustering module is used to cluster all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; extract keywords from each of the multiple clusters to obtain a cluster keyword set for each cluster, and determine the cluster center article title of each cluster;

[0028] A first determination module is used to determine, for each article, article keywords of the article based on a subject keyword set of the article and a cluster keyword set of a cluster to which the article belongs;

[0029] The second determination module is used to determine the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs.

[0030] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the article image extraction method described in the first aspect are implemented.

[0031] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the article portrait extraction method as described in the first aspect.

[0032] In a fifth aspect, an embodiment of the present invention further provides a computer program product having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the steps of the article portrait extraction method described in the first aspect.

[0033] The article portrait extraction method provided by the present invention extracts the subject probability distribution of each article in the corpus through the word frequency matrix corresponding to the corpus, and determines the subject keyword set of each article based on the subject probability distribution of each article; and clusters all articles in the corpus through the text vector matrix corresponding to the corpus to obtain multiple clusters, and then extracts the cluster keyword set of each cluster in each of the multiple clusters and determines the cluster center article title of each cluster; for each article, determines the article keywords of the article according to the subject keyword set corresponding to each article and the cluster keyword set of the cluster to which the article belongs; determines the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs, realizes the centrality of the article keywords and the cluster keyword set, improves the accuracy of subject keyword extraction, ensures the subject consistency of the recalled articles, and the intra-cluster recall of the cluster center article title, enhances the generalization of the recalled articles, and at the same time, also improves the recall quality of the recommendation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 One of the flow charts of the article portrait extraction method provided by the present invention;

[0036] Figure 2 The second flowchart of the article portrait extraction method provided by the present invention;

[0037] Figure 3 The third flow chart of the article portrait extraction method provided by the present invention;

[0038] Figure 4 The fourth flowchart of the article portrait extraction method provided by the present invention;

[0039] Figure 5 The fifth flow chart of the article portrait extraction method provided by the present invention;

[0040] Figure 6 The sixth flow chart of the article portrait extraction method provided by the present invention;

[0041] Figure 7 A schematic diagram of the structure of the article image extraction device provided by the present invention;

[0042] Figure 8 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The article portrait extraction method provided by the present invention is described in detail below through some embodiments and application scenarios in conjunction with the accompanying drawings.

[0045] The present invention provides an article portrait extraction method, which can be applied to the application scenario of a recommendation system. The subject probability distribution of each article in the corpus is extracted based on a word frequency matrix corresponding to the corpus, and a subject keyword set of each article is determined based on the subject probability distribution of each article; all articles in the corpus are clustered based on a text vector matrix corresponding to the corpus to obtain a plurality of clusters; keywords are extracted from each cluster in the plurality of clusters to obtain a cluster keyword set of each cluster, and a cluster center article title of each cluster is determined; for each article, article keywords of the article are determined based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs; an article portrait of the article is determined based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs, so as to achieve the centrality of article keywords and cluster keyword sets, improve the accuracy of subject keyword extraction, ensure the subject consistency of recalled articles, and the intra-cluster recall of cluster center article titles, enhance the generalization of recalled articles, and at the same time, improve the recall quality of the recommendation system.

[0046] Figure 1 One of the flow charts of the article image extraction method provided by the present invention is as follows: Figure 1 As shown, the method includes steps 110 to 140, wherein:

[0047] Step 110 , extracting the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determining the topic keyword set of each article based on the topic probability distribution of each article.

[0048] It should be noted that the article image extraction method provided by the present invention can be applicable to the application scenario of the recommendation system. The execution subject of the method can be an article image extraction device, for example, an electronic device, or a control module in the article image extraction device for executing the article image extraction method.

[0049] Optionally, the corpus includes multiple articles, each of which is represented by an article identifier, wherein the article identifier can be a number or other identifiers.

[0050] Optionally, the word frequency matrix refers to a vector matrix obtained by preprocessing and vectorizing multiple articles in a corpus.

[0051] Specifically, multiple articles in the corpus are preprocessed using relevant methods of natural language processing to obtain word segmentations and sentence segmentations corresponding to the multiple articles; for example, the relevant method of natural language processing may be the Jieba word segmentation method or other methods; a statistical dictionary is then used based on the word segmentations and sentence segmentations corresponding to the multiple articles to convert the word segmentations and sentence segmentations into sparse vectors, and a word frequency matrix is ​​then obtained based on the sparse vectors.

[0052] In practice, preprocessing includes removing articles without titles, content, or garbled text from the corpus, as well as removing articles with duplicate content and missing data; for example, articles without a specific release time or source; or filling in the release time for articles without a specific release time.

[0053] Optionally, the word frequency matrix is ​​input into the LDA model to obtain multiple topics output by the LDA model and the topic probability distribution of the multiple topics in each article; wherein each of the multiple topics includes at least one topic keyword, and the topic keywords in each topic may or may not have overlapping topic keywords.

[0054] Optionally, the subject keyword set refers to a set of subject keywords included in multiple subjects of each article.

[0055] Step 120, clustering all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; performing keyword extraction on each of the multiple clusters to obtain a cluster keyword set for each cluster, and determining the cluster center article title of each cluster.

[0056] Optionally, the text vector matrix refers to a vector matrix obtained by stacking the vectors corresponding to each article in the corpus.

[0057] Specifically, the titles and contents of multiple articles in the corpus are input into a bidirectional encoder representation from transformers (Bert) model to obtain a text vector matrix output by the Bert model, and the text vector matrix is ​​saved; wherein the Bert model is obtained after training based on sample articles.

[0058] It should be noted that the text vector matrix obtained according to the Bert model needs to be processed using relevant dimensionality reduction methods; the dimensionality reduction method can be the principal component analysis (PCA) method or other methods; the PCA method is a common data analysis method, which is often used for dimensionality reduction of high-dimensional data and can be used to extract the main characteristic components of the data. For example, the dimension of the text vector matrix corresponding to multiple articles in the corpus is 100*100. The PCA method is used to reduce the dimension of the text vector matrix from 100*100 to 10*10.

[0059] Optionally, the feature vector matrix obtained after the text vector matrix is ​​processed by the PCA method is input into a density-based clustering (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) model to obtain multiple clusters output by the DBSCAN model; wherein each cluster includes at least one article, and the topics and semantics of the articles in each cluster are similar.

[0060] Optionally, the cluster keyword set is used to represent the keyword set of articles whose vectors corresponding to each article in each cluster are closest to the vectors corresponding to all articles in the cluster; the cluster center article title is used to represent the title of the article in each cluster that is closest to the cluster center vector.

[0061] Step 130 : For each article, determine the article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs.

[0062] Optionally, the article keywords are obtained by weighted merging of a subject keyword set corresponding to each article in the corpus and a cluster keyword set of the cluster to which the article belongs.

[0063] Specifically, for each article in the corpus, a target weight is set for the subject keyword set of each article, for example, the target weight is a; a target weight is set for the cluster keyword set of the cluster to which each article belongs, for example, the target weight is b; and a target weight is set for the keywords of the intersection part of the subject keyword set of each article and the cluster keyword set of the cluster to which the article belongs, for example, the target weight of the keywords of the intersection part is c; the cluster keyword set of the cluster keyword set of the cluster to which each article belongs, except for the keywords of the intersection part of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, is randomly added to the keywords of the intersection part of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, to obtain the article keywords.

[0064] Step 140, determining the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs.

[0065] Optionally, the cluster center article title is encoded using one-hot encoding to obtain a serialized cluster center article title.

[0066] Optionally, the article keywords of the article and the continuous cluster center article title of the cluster to which the article belongs are updated into the article portrait, thereby determining the article portrait of the article.

[0067] The article portrait extraction method provided by the present invention extracts the subject probability distribution of each article in the corpus through the word frequency matrix corresponding to the corpus, and determines the subject keyword set of each article based on the subject probability distribution of each article; and clusters all articles in the corpus through the text vector matrix corresponding to the corpus to obtain multiple clusters, and then extracts the cluster keyword set of each cluster and determines the cluster center article title of each cluster for each cluster in the multiple clusters; for each article, determines the article keywords of the article according to the subject keyword set corresponding to each article and the cluster keyword set of the cluster to which the article belongs; determines the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs, realizes the centrality of the article keywords and the cluster keyword set, improves the accuracy of subject keyword extraction, ensures the subject consistency of the recalled articles, and the intra-cluster recall of the cluster center article title, enhances the generalization of the recalled articles, and at the same time, also improves the recall quality of the recommendation system.

[0068] Figure 2 The second flow chart of the article portrait extraction method provided by the present invention is as follows: Figure 2 As shown, the method includes steps 210 to 260, wherein:

[0069] Step 210: extracting the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus.

[0070] Optionally, the description and explanation of step 210 may refer to the description and explanation of this part in the above step 110, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0071] Step 220, based on the topic probability distribution of each article, select N maximum topic probabilities in the topic probability distribution of each article; N is an integer.

[0072] Optionally, according to the probability corresponding to each topic in the topic probability distribution of each article, N maximum topic probabilities are selected; wherein the value of N can be selected according to actual conditions; for example, when N is 3, it means selecting the top 3 topic probabilities with the largest topic probabilities in the topic probability distribution corresponding to each article.

[0073] Step 230, determining a topic keyword set for each article based on the topic keywords corresponding to the selected N maximum topic probabilities.

[0074] Optionally, based on the topic keywords corresponding to each topic in the selected N maximum topic probabilities, search in each article whether the topic keywords corresponding to each topic in the N maximum topic probabilities exist in the article, combine the topic keywords existing in the article to obtain the topic keyword set of the article, thereby determining the topic keyword set of each article.

[0075] Step 240, clustering all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; performing keyword extraction on each of the multiple clusters to obtain a cluster keyword set for each cluster, and determining the cluster center article title of each cluster.

[0076] Step 250 : For each article, determine the article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs.

[0077] Step 260, determining the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs.

[0078] Optionally, regarding the description and explanation of steps 240-260, reference may be made to the description and explanation of this part in the above steps 120-140, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0079] The article portrait extraction method provided by the present invention selects N maximum topic probabilities in the topic probability distribution of each article according to the topic probability distribution of each article, and determines the topic keyword set of each article according to the topic keywords corresponding to the N maximum topic probabilities, thereby improving the accuracy of topic keyword extraction, retaining secondary topic keywords, enhancing the generalization of the topics of recalled articles, and at the same time, improving the recall quality of the recommendation system.

[0080] Optionally, the implementation method of performing keyword extraction on each of the multiple clusters in step 240 to obtain a cluster keyword set of each cluster may include the following steps:

[0081] Step 1: Based on multiple clusters, count the number of times the keywords of each article in each cluster appear in the article.

[0082] Step 2: Combine the keywords that appear most frequently in each article to obtain a cluster keyword set for each cluster.

[0083] Specifically, all articles in the corpus are clustered according to the text vector matrix corresponding to the corpus to obtain multiple clusters, and the keyword extraction method is used to count the number of times the keywords of each article in each cluster appear in the article; wherein the keyword extraction method can be the term frequency-pseudo-text frequency index (Term Frequency-Inverse Document Frequency, TFIDF) method, which counts the keywords of each article in each cluster, or other keyword extraction methods. Then, the keywords with the largest number of occurrences in each article in each cluster are combined respectively to obtain the cluster keyword set of each cluster.

[0084] Figure 3 The third flow chart of the article portrait extraction method provided by the present invention is as follows: Figure 3 As shown, the method includes steps 310 to 360, wherein:

[0085] Step 310: extract the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determine the topic keyword set of each article based on the topic probability distribution of each article.

[0086] Step 320: cluster all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; extract keywords from each of the multiple clusters to obtain a cluster keyword set for each cluster.

[0087] Optionally, regarding the description and explanation of steps 310-320, reference may be made to the description and explanation of this part in steps 110-120, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0088] Step 330 : Based on the multiple clusters, respectively calculate the distance between the vector corresponding to each article in each cluster and the vector corresponding to all articles in each cluster, and determine the cluster center vector of each cluster.

[0089] Optionally, a method for calculating cosine distance is used to calculate the distance between the vector corresponding to each article in each cluster and the vector corresponding to all articles in each cluster, and the vector with the smallest distance is the cluster center vector of the cluster; wherein the vector corresponding to all articles in each cluster is obtained by stacking the vectors corresponding to each article in each cluster.

[0090] Step 340: Determine the cluster center article title of each cluster based on the cluster center vector.

[0091] Optionally, the title of the article corresponding to the cluster center vector is used as the cluster center article title of the cluster, thereby determining the cluster center article title of each cluster.

[0092] Step 350 : For each article, determine the article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs.

[0093] Step 360, determining the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs.

[0094] Optionally, regarding the description and explanation of steps 350-360, reference may be made to the description and explanation of this part in steps 130-140, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0095] The article portrait extraction method provided by the present invention calculates the distance between the vector corresponding to each article in each cluster and the vector corresponding to all articles in each cluster according to multiple clusters, and the vector with the smallest distance is the cluster center vector. Then, according to the titles of the articles corresponding to the cluster center vectors of each cluster, the cluster center article titles of each cluster are determined. According to the intra-cluster recall of the cluster center article titles, the generalization of the recalled articles is enhanced, and at the same time, the recall quality of the recommendation system is also improved.

[0096] Optionally, the specific implementation of the above step 340 may include the following steps:

[0097] Step 3: Based on the cluster center vector, determine the article corresponding to the vector closest to the cluster center vector in each cluster as the cluster center article.

[0098] Step 4: Based on the titles of the cluster center articles, determine the cluster center article titles of each cluster.

[0099] Specifically, according to the cluster center vector of each cluster, the article in each cluster that is closest to the cluster center vector of the cluster is used as the cluster center article of the cluster; and then the article title of the cluster center article of each cluster is used as the cluster center article title of each cluster.

[0100] Figure 4 The fourth flow chart of the article image extraction method provided by the present invention is as follows: Figure 4 As shown, the method includes steps 410 to 450, wherein:

[0101] Step 410: extract the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determine the topic keyword set of each article based on the topic probability distribution of each article.

[0102] Step 420, clustering all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; performing keyword extraction on each of the multiple clusters to obtain a cluster keyword set for each cluster, and determining the cluster center article title of each cluster.

[0103] Optionally, regarding the description and explanation of steps 410-420, reference may be made to the description and explanation of this part in steps 110-120, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0104] Step 430 : determining the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs.

[0105] Optionally, for each article, based on the subject keyword set of each article and the cluster keyword set of the cluster to which the article belongs, determine the keywords that are the same between the subject keyword set of each article and the cluster keyword set of the cluster to which the article belongs, that is, determine the intersection of the subject keyword set of each article and the cluster keyword set of the cluster to which the article belongs.

[0106] Step 440 , determining article keywords of the article based on the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, and a preset number of cluster keywords in the cluster keyword set excluding the intersection.

[0107] Optionally, the preset number refers to the number of cluster keywords randomly added to the intersection of the cluster keyword set excluding the intersection; wherein the preset number can be any number of keywords, for example, the preset number can be 3 cluster keywords in the cluster keyword set excluding the intersection, which is set according to actual conditions.

[0108] Step 450, determining the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs.

[0109] The article portrait extraction method provided by the present invention determines the intersection of the subject keyword set corresponding to each article and the cluster keyword set of the cluster to which the article belongs, based on the subject keyword set corresponding to the article and the cluster keyword set of the cluster to which the article belongs; selects a preset number of cluster keywords in the cluster keyword set except the intersection, and merges them with the keywords of the intersection part to determine the article keywords of the article, thereby achieving the centralization of the article keywords and the cluster keyword set, improving the accuracy of subject keyword extraction, ensuring the subject consistency of the recalled articles, and the extraction of the cluster keyword set also enhances the generalization of the recalled articles, and at the same time, improves the recall quality of the recommendation system.

[0110] Figure 5 The fifth flow chart of the article image extraction method provided by the present invention is as follows: Figure 5 As shown, the method includes steps 501 to 512, wherein:

[0111] Step 501, data preprocessing. Use manual data cleaning to remove articles without titles, content, or garbled text in the corpus, as well as articles with duplicate content. At the same time, fill in articles with missing relevant data, such as filling in articles without release time or release source; use natural language processing related methods to preprocess data for all articles in the corpus; for example, use the Jieba method to segment all articles in the corpus into words and sentences to obtain the words and sentences corresponding to the articles in the corpus.

[0112] Step 502, data vectorization. All articles in the corpus after preprocessing are input into the Bert model, and the Bert model outputs a text vector matrix corresponding to all articles in the corpus; wherein the Bert model is obtained after training based on the sample articles to be trained; for example, there are 100 articles in the corpus, and the Bert model is set to output a 1*100 vector, then the 100 articles in the corpus are input into the Bert model in sequence, and the Bert model outputs 100 1*100 vectors in sequence, and the 100 1*100 vectors are stacked to obtain a 100*100 text vector matrix.

[0113] Optionally, the word segments and sentence segments obtained after preprocessing all articles in the corpus are converted into sparse vectors using a statistical dictionary, and then a word frequency matrix is ​​obtained based on the sparse vectors.

[0114] Step 503: LDA model topic analysis: Input the word frequency matrix into the LDA model to obtain multiple topics corresponding to all articles output by the LDA model and the topic probability distribution of each article.

[0115] Specifically, the word frequency matrix corresponding to the corpus is input into the LDA model, and the LDA model outputs multiple topics and the topic probability distribution of multiple topics in each article; for example, the LDA model outputs 5 topics, each topic corresponds to multiple topic keywords; among them, Topic 1: [training, institution, class, course]; Topic 2: [wedding, wedding, photographer, laid-off]; Topic 3: [tertiary industry, 100 million yuan, first quarter, growth rate]; Topic 4: [World Health Organization, public, consequences]; Topic 5: [strawberry, hygiene, farmers, greenhouse, fruit farmers, maturity]; and the topic probability distribution of each topic corresponding to each article, among which the probabilities of Topic 1 to Topic 5 in Article 1, the probabilities of Topic 1 to Topic 5 in Article 2, the probabilities of Topic 1 to Topic 5 in Article 3, etc.

[0116] Step 504, extract the topic distribution of the article. According to the topic probability distribution of each article, select the N topic probabilities with the largest probability in the topic probability distribution of each article. For example, select the first three largest topic probabilities in the topic probability distribution of each article. According to the probability distribution of each topic in article 1, determine that the topic probabilities corresponding to topic 1, topic 2 and topic 3 are the largest, thereby determining the topic distribution of article 1 as topic 1, topic 2 and topic 3; according to the probability distribution of each topic in article 2, determine that the topic probabilities corresponding to topic 1, topic 3 and topic 4 are the largest, thereby determining the topic distribution of article 2 as topic 1, topic 3 and topic 4; and so on, determine the topic distribution of each article.

[0117] Step 505, determine the theme keyword set of each article. Determine the theme keyword set of each article according to the theme probability distribution of each article; for example, according to the probability distribution of each theme in article 1, determine that the theme probability corresponding to theme 1, theme 2 and theme 3 is the largest, then according to the theme keywords corresponding to theme 1, theme 2 and theme 3, respectively match the theme keywords corresponding to theme 1, theme 2 and theme 3 with the keywords in article 1, and determine the theme keyword set of article 1 according to the theme keywords corresponding to theme 1, theme 2 and theme 3 respectively and the same theme keywords in the keywords in article 1; and so on, determine the theme keyword set of each article.

[0118] Step 506: Cluster all articles in the corpus to obtain multiple clusters. First, use the PCA method to reduce the dimension of the text vector matrix. For example, if the dimension of the text vector matrix is ​​100*100, use the PCA method to reduce the dimension and obtain a eigenvector matrix with a dimension of 10*10; input the eigenvector matrix into the DBSCAN model to obtain multiple article clusters output by the DBSCAN model; each cluster includes at least one article, and the articles in each cluster have similar topics and semantics.

[0119] Step 507: Obtain the number of times the keyword of each article appears in the article. Use the TFIDF keyword extraction method to count the number of times the keyword of each article in each cluster appears in the article.

[0120] Step 508: extract the cluster keyword set of each cluster. According to the number of times the keywords of each article in each cluster appear in the article, match the keywords with the largest number of appearances in each article, combine them, and obtain the cluster keyword set of each cluster.

[0121] Step 509, determine the cluster center article title. According to the multiple article clusters output by the DBSCAN model, calculate the distance between the vector corresponding to each article in each cluster and the vector corresponding to all articles in each cluster, and the vector with the closest distance is the cluster center vector of each cluster; then, according to the cluster center vector, determine the article corresponding to the vector with the closest distance to the cluster center vector in each cluster as the cluster center article; use the title of the cluster center article of each cluster as the cluster center article title of each cluster.

[0122] Step 510: Serialize the cluster center article titles. According to the cluster center article titles of each cluster, use the One-Hot encoding method to encode the cluster center article titles of each cluster to obtain serialized cluster center article titles.

[0123] Step 511, determine the article keywords. Set target weights for the subject keyword set of each article and the cluster keyword set of each cluster, for example, set the target weight of the subject keyword set to a, and the target weight of the cluster keyword set to b; determine the intersection of the subject keyword set of each article and the cluster keyword set of each cluster according to the subject keyword set of each article and the cluster keyword set of each cluster, and set the target weight of the keywords in the intersection to c. Randomly add a preset number of cluster keywords in the cluster keyword set of each cluster, except for the intersection of the subject keyword set of each article and the cluster keyword set of each cluster, to the intersection of the subject keyword set of each article and the cluster keyword set of each cluster, so as to determine the article keywords of each article.

[0124] Step 512, determining the article portrait. The article keywords and the article title of the continuous cluster center are updated to the article portrait, that is, the article portrait of the article is determined according to the article keywords and the article title of the continuous cluster center.

[0125] The article portrait extraction method provided by the present invention obtains a word frequency matrix and a text vector matrix corresponding to all the articles in the corpus by preprocessing and vectorizing all the articles in the corpus; extracts the subject probability distribution of each article in the corpus according to the word frequency matrix, and determines the subject keyword set of each article based on the subject probability distribution of each article; and clusters all the articles in the corpus according to the text vector matrix to obtain multiple clusters, and then extracts keywords from each cluster in the multiple clusters to obtain the cluster keyword set of each cluster, and determines the cluster center article title of each cluster; determines the article keywords of the article according to the subject keyword set corresponding to each article and the cluster keyword set of the cluster to which the article belongs; determines the article portrait of the article according to the article keywords of the article and the cluster center article title of the cluster to which the article belongs, realizes the centrality of the article keywords and the cluster keyword set, improves the accuracy of subject keyword extraction, ensures the subject consistency of the recalled articles, and the intra-cluster recall of the cluster center article title, enhances the generalization of the recalled articles, and also improves the recall quality of the recommendation system.

[0126] Figure 6 The sixth flow chart of the article portrait extraction method provided by the present invention is as follows: Figure 6 As shown, this embodiment takes a sports article as an example to describe in detail the article portrait extraction method provided by the present invention, wherein:

[0127] Step 1: Use manual data cleaning to remove articles without titles, content, or garbled text in the corpus, as well as articles with duplicate content. At the same time, fill in articles with missing relevant data, for example, fill in articles with no publication time or source. Use natural language processing methods to preprocess data for all articles in the corpus. For example, use the jieba method to segment all articles in the corpus into words and sentences to obtain the words and sentences corresponding to the articles in the corpus.

[0128] Step 2: Input all articles in the corpus after preprocessing into the Bert model to obtain the text vector matrix corresponding to all articles in the corpus output by the Bert model; wherein the Bert model is obtained after training based on the sample articles to be trained; for example, there are 100 articles in the corpus, and the Bert model is set to output a 1*100 vector. Then, the 100 articles in the corpus are input into the Bert model in sequence, and the Bert model outputs 100 1*100 vectors in sequence. The 100 1*100 vectors are stacked to obtain a text vector matrix with a dimension of 100*100. The word segmentation and sentence segmentation obtained after preprocessing all articles in the corpus are converted into sparse vectors using a statistical dictionary, and then the word frequency matrix is ​​obtained based on the sparse vectors.

[0129] Step 3, input the word frequency matrix into the LDA model to obtain multiple topics corresponding to all articles output by the LDA model and the topic probability distribution of each article; wherein, the target number of topics to be generated is set for the LDA model, for example, the LDA model is set to generate 100 topics, then the LDA model outputs 100 topics and the topic probability distribution of each article; wherein, each topic includes multiple topic keywords.

[0130] Step 4. According to the topic probability distribution of each article, select the N topic probabilities with the largest probability in the topic probability distribution of each article. For example, select the top 3 largest topic probabilities in the topic probability distribution of each article. According to the probability distribution of each topic in Article 1, determine that the topic probabilities corresponding to Topic 1, Topic 2 and Topic 3 are the largest, thereby determining the topic distribution of Article 1 as Topic 1, Topic 2 and Topic 3; according to the probability distribution of each topic in Article 2, determine that the topic probabilities corresponding to Topic 1, Topic 3 and Topic 4 are the largest, thereby determining the topic distribution of Article 2 as Topic 1, Topic 3 and Topic 4; and so on, determine the topic distribution of each article.

[0131] Step 5, match the keywords of each article according to the topic keywords corresponding to the top three largest topic probabilities in the topic probability distribution of each article, and determine the topic keyword set of each article; it should be noted that the topic keywords of each topic in the topic keyword set of each article may overlap; for example, the topic keyword set of the sports article is obtained according to the identifier of the sports article, and the topic keyword set of the article is [[football, Champions League, free kick, team], [team, league], [Italy]], where the keyword "team" exists in two topics.

[0132] Step 6: Cluster all articles in the corpus to obtain multiple clusters. The text vector matrix obtained by the Bert model is reduced to 100 dimensions using the PCA method; the 100-dimensional text vector matrix is ​​then input into the DBSCAN model for clustering to obtain multiple clusters output by the DBSCAN model.

[0133] Step 7, using the TFIDF keyword extraction method, count the number of times the keywords of each article in each cluster appear in the article, match the keywords with the largest number of appearances in each article, and combine them to obtain the cluster keyword set of each cluster. For example, according to the identification of the sports article, the cluster keyword set of the cluster to which the sports article belongs is obtained, and the cluster keyword set is [football, Champions League, free kick, national team, substitute, Tokyo Olympics, head coach, athlete, gold medal].

[0134] Step 8. According to the multiple article clusters output by the DBSCAN model, calculate the distance between the vector corresponding to each article in each cluster and the vector corresponding to all articles in each cluster. The vector with the shortest distance is the cluster center vector of each cluster. Then, according to the cluster center vector, determine the article corresponding to the vector with the shortest distance to the cluster center vector in each cluster as the cluster center article. Use the title of the cluster center article of each cluster as the cluster center article title of each cluster.

[0135] Step 9: Encode the cluster center article titles of each cluster using the One-Hot encoding method according to the cluster center article titles of each cluster to obtain a continuous cluster center article title.

[0136] Step 10, set target weights for the subject keyword set of each article and the cluster keyword set of each cluster, for example, set the target weight of the subject keyword set to 1 and the target weight of the cluster keyword set to 2; determine the intersection of the subject keyword set of each article and the cluster keyword set of each cluster according to the subject keyword set of each article and the cluster keyword set of each cluster, and set the target weight of the keywords in the intersection to 3. Randomly select 3 cluster keywords from the cluster keyword set of each cluster except the intersection of the subject keyword set of each article and the cluster keyword set of each cluster and add them to the intersection of the subject keyword set of each article and the cluster keyword set of each cluster, so as to determine the article keywords of each article.

[0137] Step 11, update the article keywords and the article title of the continuous cluster center to the article portrait of the sports article.

[0138] It should be noted that if the user clicks on the sports article, the recommendation system will update the article profile of the sports article to the user profile. When the recommendation system recommends articles to the user, it will recall relevant articles in the corpus based on the encoding of the article keywords and the cluster center article title, and recommend the obtained list of articles to be recommended to the user.

[0139] The article image extraction device provided by the present invention is described below. The article image extraction device described below and the article image extraction method described above can be referenced to each other.

[0140] Figure 7 The structural diagram of the article image extraction device provided by the present invention is as follows: Figure 7 As shown, the article portrait extraction device 700 includes: an extraction module 701, a clustering module 702, a first determination module 703, and a second determination module 704; wherein,

[0141] An extraction module 701 is used to extract the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determine the topic keyword set of each article based on the topic probability distribution of each article;

[0142] The clustering module 702 is used to cluster all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; extract keywords from each of the multiple clusters to obtain a cluster keyword set of each cluster, and determine the cluster center article title of each cluster;

[0143] A first determination module 703 is used to determine, for each article, article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs;

[0144] The second determination module 704 is used to determine the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs.

[0145] The article portrait extraction device provided by the present invention extracts the subject probability distribution of each article in the corpus through the word frequency matrix corresponding to the corpus, and determines the subject keyword set of each article based on the subject probability distribution of each article; and clusters all articles in the corpus through the text vector matrix corresponding to the corpus to obtain multiple clusters, and then extracts the cluster keyword set of each cluster and determines the cluster center article title of each cluster for each cluster in the multiple clusters; for each article, determines the article keywords of the article according to the subject keyword set corresponding to each article and the cluster keyword set of the cluster to which the article belongs; determines the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs, realizes the concentration of article keywords and cluster keyword sets, improves the accuracy of subject keyword extraction, ensures the subject consistency of recalled articles and the intra-cluster recall of cluster center article titles, enhances the generalization of recalled articles, and at the same time, improves the recall quality of the recommendation system.

[0146] Optionally, the extraction module 701 is specifically configured to:

[0147] Based on the topic probability distribution of each article, select the N largest topic probabilities in the topic probability distribution of each article; N is an integer;

[0148] Based on the topic keywords corresponding to the selected N maximum topic probabilities, a topic keyword set for each article is determined.

[0149] Optionally, the clustering module 702 is specifically configured to:

[0150] Based on the multiple clusters, the number of times a keyword of each article in each cluster appears in the article is counted; the keywords with the largest number of appearances in each article are combined to obtain a cluster keyword set of each cluster.

[0151] Optionally, the clustering module 702 is specifically configured to:

[0152] Based on the multiple clusters, respectively calculating the distance between the vector corresponding to each article in each cluster and the vector corresponding to all articles in each cluster, and determining the cluster center vector of each cluster;

[0153] Based on the cluster center vector, the cluster center article title of each cluster is determined.

[0154] Optionally, the clustering module 702 is specifically configured to:

[0155] Based on the cluster center vector, determine in each cluster that the article corresponding to the vector closest to the cluster center vector is the cluster center article;

[0156] Based on the titles of the cluster center articles, the cluster center article titles of each cluster are determined.

[0157] Optionally, the first determining module 703 is specifically configured to:

[0158] Based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, determining the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs;

[0159] The article keywords of the article are determined based on the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, and a preset number of cluster keywords in the cluster keyword set except the intersection.

[0160] Figure 8 A schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 8As shown, the electronic device 800 may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the following article portrait extraction method: extract the subject probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determine the subject keyword set of each article based on the subject probability distribution of each article; cluster all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; extract keywords from each cluster in the multiple clusters to obtain the cluster keyword set of each cluster, and determine the cluster center article title of each cluster; for each article, determine the article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs; determine the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs.

[0161] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0162] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the following article portrait extraction method is implemented: extracting the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determining the topic keyword set of each article based on the topic probability distribution of each article; clustering all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; performing keyword extraction on each of the multiple clusters to obtain a cluster keyword set of each cluster, and determining the cluster center article title of each cluster; for each article, determining the article keywords of the article based on the topic keyword set of the article and the cluster keyword set of the cluster to which the article belongs; determining the article portrait of the article based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs.

[0163] On the other hand, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the following article portrait extraction method is implemented: based on the word frequency matrix corresponding to the corpus, the topic probability distribution of each article in the corpus is extracted, and the topic keyword set of each article is determined based on the topic probability distribution of each article; based on the text vector matrix corresponding to the corpus, all articles in the corpus are clustered to obtain multiple clusters; keywords are extracted for each cluster in the multiple clusters to obtain a cluster keyword set of each cluster, and the cluster center article title of each cluster is determined; for each article, the article keywords of the article are determined based on the topic keyword set of the article and the cluster keyword set of the cluster to which the article belongs; based on the article keywords of the article and the cluster center article title of the cluster to which the article belongs, the article portrait of the article is determined.

[0164] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting article images, characterized in that: The method comprises: Extracting the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determining the topic keyword set of each article based on the topic probability distribution of each article; Clustering all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; extracting keywords from each of the multiple clusters to obtain a cluster keyword set for each cluster, and determining the cluster center article title of each cluster; For each article, determining article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs; Determine an article profile of the article based on article keywords of the article and the cluster center article title of the cluster to which the article belongs; The step of extracting keywords from each of the plurality of clusters to obtain a cluster keyword set of each cluster includes: Based on the multiple clusters, counting the number of times a keyword of each article in each cluster appears in the article; combining the keywords with the largest number of appearances in each article to obtain a cluster keyword set for each cluster; The step of determining the cluster center article title of each cluster includes: Based on the multiple clusters, respectively calculating the distance between the vector corresponding to each article in each cluster and the vector corresponding to all articles in each cluster, and determining the cluster center vector of each cluster; Based on the cluster center vector, determine the cluster center article title of each cluster; For each article, determining the article keywords of the article based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs includes: Based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, determining the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs; The article keywords of the article are determined based on the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, and a preset number of cluster keywords in the cluster keyword set except the intersection.

2. The article image extraction method according to claim 1, characterized in that: Determining the subject keyword set of each article based on the subject probability distribution of each article includes: Based on the topic probability distribution of each article, select the N largest topic probabilities in the topic probability distribution of each article; N is an integer; Based on the topic keywords corresponding to the selected N maximum topic probabilities, a topic keyword set for each article is determined.

3. The article image extraction method according to claim 1, characterized in that: The step of determining the cluster center article title of each cluster based on the cluster center vector comprises: Based on the cluster center vector, determine in each cluster that the article corresponding to the vector closest to the cluster center vector is the cluster center article; Based on the titles of the cluster center articles, the cluster center article titles of each cluster are determined.

4. An article image extraction device, characterized in that: The device comprises: An extraction module, used to extract the topic probability distribution of each article in the corpus based on the word frequency matrix corresponding to the corpus, and determine the topic keyword set of each article based on the topic probability distribution of each article; A clustering module is used to cluster all articles in the corpus based on the text vector matrix corresponding to the corpus to obtain multiple clusters; extract keywords from each of the multiple clusters to obtain a cluster keyword set for each cluster, and determine the cluster center article title of each cluster; A first determination module is used to determine, for each article, article keywords of the article based on a subject keyword set of the article and a cluster keyword set of a cluster to which the article belongs; A second determination module is used to determine an article portrait of the article based on article keywords of the article and a cluster center article title of the cluster to which the article belongs; Clustering module, specifically used for: Based on the multiple clusters, counting the number of times a keyword of each article in each cluster appears in the article; combining the keywords with the largest number of appearances in each article to obtain a cluster keyword set for each cluster; Based on the multiple clusters, respectively calculating the distance between the vector corresponding to each article in each cluster and the vector corresponding to all articles in each cluster, and determining the cluster center vector of each cluster; Based on the cluster center vector, determine the cluster center article title of each cluster; The first determination module is specifically used for: Based on the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, determining the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs; The article keywords of the article are determined based on the intersection of the subject keyword set of the article and the cluster keyword set of the cluster to which the article belongs, and a preset number of cluster keywords in the cluster keyword set except the intersection.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the article image extraction method as described in any one of claims 1 to 3 are implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the article image extraction method as described in any one of claims 1 to 3 are implemented.

7. A computer program product having executable instructions stored thereon, characterized in that: When the instruction is executed by the processor, the processor implements the steps of the article image extraction method as described in any one of claims 1 to 3.

Images