A network article and theme relevance analysis method based on data imaging

Abstract

The application discloses a network article and theme correlation analysis method based on data imaging, comprising: obtaining text data to be processed; converting the text data to be processed into a first feature map based on a pre-constructed mapping table; and counting the positions of each word group appearing in the text data, and constructing a second feature map based on the appearing positions of each word group; fusing the first feature map and the second feature map as two-channel data, and inputting into a trained image classification model to complete correlation judgment by using the trained image classification model. The embodiment of the application first extracts effective features of an article by using natural language preprocessing technology, then performs feature imaging, and finally performs classification and calculation of the correlation with a target theme by using a convolutional neural network, so that accurate, fast and low-power data analysis is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a network article and theme relevance analysis method based on data image. BACKGROUND

[0002] At present, network article or news topic classification methods include classic cosine classification, matrix singular value decomposition algorithm, and some natural language processing-based methods such as recurrent neural network (RNN).

[0003] In the traditional method, the singular value decomposition method needs large memory or distributed computing, and the classification result is rough, and the cosine classification algorithm needs multiple iterations, is slow, and the feature utilization is not sufficient, such as the correlation between each dimension of the vector.

[0004] The AI-based method is mainly based on recurrent neural network (RNN) and graph neural network (Transformer), and the processing process is as shown in Figure 1 The network structure is GRU, LSTM, etc.

[0005] The prior art scheme has a complex network structure to capture long-range dependence features, low parallelism, and relatively slow speed. Meanwhile, the prior art scheme needs to redesign the network structure to be compatible with multiple features, and the network compatibility is low. The full connection layer in the network structure has a large amount of calculation, which is not conducive to network optimization. SUMMARY

[0006] The embodiment of the present application provides a network article and theme relevance analysis method based on data imageization, which first extracts effective features of the article by using natural language preprocessing technology, then performs feature imageization, and finally uses a convolutional neural network to classify and calculate the relevance to the target theme, to realize accurate, fast and low-power data analysis.

[0007] The embodiment of the present application provides a network article and theme relevance analysis method based on data imageization, which includes the following steps:

[0008] Obtain text data to be processed;

[0009] Based on a pre-constructed mapping table, the text data to be processed is converted into a first feature map; and

[0010] The positions of each word group appearing in the text data are counted, and a second feature map is constructed based on the appearing positions of each word group;

[0011] The first feature map and the second feature map are fused as two-channel data, and input into a trained image classification model, so as to complete relevance judgment by using the trained image classification model.

[0012] Optionally, the mapping table is constructed in advance in the following manner:

[0013] Commonly used phrases are taken as the training set, and the commonly used phrases are arranged in rows and columns;

[0014] The inverse text frequency index of each phrase is calculated, satisfying: Where D is the total number of web pages, D ij is the number of web pages corresponding to the i-th row and j-th column phrase;

[0015] According to the arrangement of the phrases, an inverse text frequency index table (IDF table) is generated;

[0016] Common topics and keywords are set as article classification targets;

[0017] The mapping relationship between the IDF table and the article classification targets is constructed, that is, the mapping table.

[0018] Optionally, the image classification model is trained in advance in the following manner:

[0019] The one-hot encoding form is used, and auxiliary correction is performed after encoding to add labels to each phrase of the training set.

[0020] Optionally, the image classification model is also trained in advance in the following manner:

[0021] The training set with added labels is used to train the image classification model, and the labels are corrected according to the trained model, and the training is repeated.

[0022] Optionally, based on the mapping table constructed in advance, the text vector is converted into a first feature map, comprising:

[0023] A two-dimensional matrix of text term frequency in the text data to be processed is calculated, satisfying:

[0024]

[0025] Where M ij is the number of occurrences of the i-th row and j-th column phrase in the text data, and M is the total number of phrases in the text data;

[0026] Normalization is performed:

[0027] TF.IDF ij = TF ij *IDF ij

[0028] After normalization, the first feature map is obtained.

[0029] Optionally, the positions of each phrase in the text data are counted, and a second feature map is constructed based on the positions of each phrase.

[0030] The positions of each phrase in the text data are configured with corresponding labels, and a second feature map is constructed according to the order of the positions of each phrase in the text data.

[0031] The embodiment of the present application also proposes a network article and theme relevance analysis device, comprising a processor and a memory, the memory stores a computer program, and the computer program is executed by the processor to realize the steps of the network article and theme relevance analysis method based on data imaging as described above.

[0032] The embodiment of the present application also proposes a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to realize the steps of the network article and theme relevance analysis method based on data imaging as described above.

[0033] The embodiment of the present application first extracts the effective features of the article by using natural language preprocessing technology, then performs feature imaging, and finally uses convolutional neural network to classify and calculate the relevance to the target theme, realizing accurate, fast and low-power data analysis.

[0034] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, which can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0035] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered as limiting the present application. Moreover, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0036] Figure 1 for the classification process based on natural language processing of the prior art;

[0037] Figure 2 for the data imaging of the network article and theme relevance analysis method of the present application;

[0038] Figure 3 for the label encoding form of the network article and theme relevance analysis method of the present application. DETAILED DESCRIPTION

[0039] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0040] The embodiment of the present application provides a network article and theme relevance analysis method based on data imaging, comprising the following steps:

[0041] In step S101, text data to be processed is obtained. Specifically, in some examples, distributed crawler technology can be used to obtain article data (text data to be processed).

[0042] In step S102, the text data to be processed is converted into a first feature map based on a pre-constructed mapping table. The present application converts vectors into two-dimensional pictures. The embodiment of the present application converts vectorization into imaging, that is, each dimension of the word vector is arranged in a two-dimensional manner (such as Figure 2 ), into a picture, and the projection value of each dimension is the gray value of each pixel point of the image, and then input into the CNN network.

[0043] In step S103, the positions of each word group appearing in the text data are counted, and a second feature map is constructed based on the positions of each word group.

[0044] In step S104, the first feature map and the second feature map are fused as two-channel data, and input into a trained image classification model to complete relevance judgment using the trained image classification model. If there are multiple forms of features, they are combined into a multi-channel picture, similar to the multi-channel (RGB A channel) in a color image, to facilitate the fusion of multiple features.

[0045] The embodiment of the present application first extracts effective features of the article using natural language preprocessing technology, then performs feature imaging, and finally uses a convolutional neural network to classify and calculate the relevance to the target theme, realizing accurate, fast, and low-power data analysis.

[0046] In some embodiments, the mapping table is pre-constructed in the following manner:

[0047] Common word groups are taken as the training set, and the common word groups are arranged in rows and columns, for example, the common word groups can be arranged in rows and columns. The present scheme takes 65536 groups as an example, and arranges them into a 256(row)*256(column) array.

[0048] The inverse text frequency index of each word group is calculated, which satisfies: wherein log() is the logarithm with base 2, D is the total number of web pages, D ij is the number of web pages corresponding to the i-th row and j-th column word group.

[0049] According to the arrangement mode of the word group, an inverse text frequency index table (IDF table) is generated.

[0050] Common topics and keywords are set as article classification targets, and in some examples, the article classification targets can also be numbered.

[0051] A mapping relationship between the IDF table and the article classification targets is constructed, that is, the mapping table.

[0052] In some embodiments, the image classification model is pre-trained in the following manner:

[0053] The one-hot encoding form is used, and auxiliary correction is performed after encoding to add labels to each word group of the training set. In the embodiments of the present application, as shown in Figure 3 some articles can be related to multiple topics. In specific examples, a natural language classification model (Transformer, etc.) with large (slow) speed and good classification effect can be used to generate a small amount of manual cleaning and correction, which belongs to the Teacher-Student mode in the model compression (knowledge distillation) method. A complex and large model is used as a Teacher, and the Student model structure is relatively simple. The Teacher is used to assist the training of the Student model.

[0054] In some embodiments, the image classification model is also pre-trained in the following manner:

[0055] The training set with added labels is used to train the image classification model, and the labels are corrected according to the trained model, and the training is repeated. In some specific examples, ResNet network can be used as the image classification model. First, the ResNet network method is used for pre-training, then the labels are re-corrected according to the trained model, and the ResNet network is used again to complete the training.

[0056] In some embodiments, based on the pre-constructed mapping table, the text vector is converted into a first feature map, which includes:

[0057] A two-dimensional matrix of text word frequency in the text data to be processed is calculated, which satisfies:

[0058]

[0059] wherein, M ijis the number of occurrences of the word group corresponding to the ith row and jth column in the text data, and M is the total number of word groups in the text data;

[0060] Normalization is performed:

[0061] TF.IDF ij = TF ij *IDF ij

[0062] After normalization, a first feature map is obtained.

[0063] In some embodiments, the positions of the word groups in the text data are counted, and a second feature map is constructed based on the positions of the word groups. In some specific examples, for the training process, the positions of the 65536 words in the article can be marked in the following manner, for example, 0 for no occurrence, 1 for occurrence in the title, 2 for occurrence in the abstract, 3 for occurrence in the main text, 4 for occurrence in both the title and the abstract, 5 for occurrence in both the title and the main text, 6 for occurrence in both the abstract and the main text, and 7 for occurrence in all three places. The mark values are filled into a two-dimensional array to become a two-dimensional map, which is normalized to become the second channel of the input data. The foregoing marking manner is an exemplary marking manner, and is not limited thereto.

[0064] For the inference process, the position P ij of each word group (ith row and jth column) in the article can be counted, and P ij has a value of 0, 1, 2, 3, 4, 5, 6, or 7. In an exemplary configuration, 0 indicates that the word group does not occur in the article, 1 indicates that the word group occurs only in the title, and so on. Thus, P ij forms a two-dimensional map.

[0065] Subsequently, data imaging is performed, and a column of numerical values representing the relevance to each topic is obtained by network operation. Subsequent processing can be flexible according to requirements, for example, articles related to both topic 1 and topic 6 can be selected, articles related to topic 1 but not related to topic 6 can be selected, and whether two articles are similar can be compared, and so on.

[0066] The present scheme adopts natural language data imaging and multi-feature fusion methods, and converts natural language processing into image processing, which can be captured by a simple network with multiple levels of features. Compared with a natural language model, the present scheme has the characteristics of fast processing speed and low running cost.

[0067] This application also proposes a device for analyzing the relevance of online articles to topics, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the aforementioned data visualization-based method for analyzing the relevance of online articles to topics.

[0068] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned method for analyzing the relevance of online articles and topics based on data visualization.

[0069] This application first uses natural language preprocessing technology to extract effective features from the article, then performs feature visualization, and finally uses a convolutional neural network for classification and calculation of the relevance to the target topic, thereby achieving accurate, fast, and low-power data analysis.

[0070] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0071] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0073] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A method for analyzing the relevance of online articles to specific topics based on data visualization, characterized in that, Includes the following steps: Obtain the text data to be processed; Based on a pre-built mapping table, the text data to be processed is transformed into a first feature map; and The positions of each word group in the text data are counted, and a second feature map is constructed based on the positions of each word group. The first feature map and the second feature map are fused as two-channel data and input into the trained image classification model to complete the correlation judgment using the trained image classification model. The mapping table is pre-built in the following manner: Commonly used phrases are selected as the training set, and the commonly used phrases are arranged in rows and columns; Calculate the inverse text frequency index of each phrase, satisfying: Where D is the total number of web pages, For the first i Line 1 j List the number of web pages corresponding to each phrase; Based on the arrangement of word groups, an Inverse Text Frequency Index (IDF) table is generated accordingly; Set up common themes and keywords as article categorization targets; The mapping relationship between the IDF table and the article classification target is constructed, which is the mapping table; Based on a pre-built mapping table, transforming the text vector into a first feature map includes: Calculate the two-dimensional word frequency matrix of the text data to be processed, satisfying: in, For the first i Line 1 j The column corresponds to the number of times the phrase appears in the text data, where M is the total number of phrases in the text data; Normalize: After normalization, the first feature map is obtained; The process of identifying the positions of each word group in the text data and constructing a second feature map based on these positions includes: Configure corresponding tags for the positions of each word group in the text data, and construct a second feature map according to the order in which each word group appears in the text data.

2. The method for analyzing the relevance of online articles and topics based on data visualization as described in claim 1, characterized in that, The image classification model is pre-trained using the following method: One-hot encoding is used, and auxiliary corrections are performed after encoding to add labels to each phrase in the training set.

3. The method for analyzing the relevance of online articles and topics based on data visualization as described in claim 2, characterized in that, The image classification model was also pre-trained using the following method: The image classification model is trained using the labeled training set, and the labels are corrected based on the trained model. The training is then repeated.

4. A device for analyzing the relevance of online articles to specific topics, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the steps of the data visualization-based network article and topic relevance analysis method as described in any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data visualization-based network article and topic relevance analysis method as described in any one of claims 1 to 3.

Images