Text monitoring method and device based on hybrid word vector encoding

Abstract

The application discloses a text monitoring method and device based on mixed word vector coding; wherein the method comprises: crawling public opinion text; using a BERT model to encode the public opinion text to obtain a word vector corresponding to each character; performing word segmentation on the public opinion text to obtain a plurality of words; searching for a first word vector corresponding to the word from a preset dictionary; the first word vector is a spatial anisotropy improved word vector obtained by using a linear discriminant analysis method to perform word vector clustering on an initial word vector according to categories; obtaining a second word vector corresponding to the word according to the word vector; merging the first word vector and the second word vector corresponding to the word to obtain an improved word vector; and performing public opinion monitoring based on the improved word vector, which solves the problems of narrow word vector coding space and obvious anisotropy in the prior art when using a BERT model for word vector coding.

Description

Technical Field

[0001] This invention belongs to the field of electrical data processing technology, specifically relating to a text monitoring method and apparatus based on hybrid word vector encoding. Background Technology

[0002] In the context of big data, most public opinion analysis platforms use natural language processing algorithms combined with deep learning algorithms to analyze text, facilitating efficient and timely analysis of large sample data. The most crucial step is feature extraction from natural language text. From static word vectors (Word2Vec, GloVe) to the BERT text encoding model based on the Transformer structure, text feature extraction has undergone a series of developments and improvements.

[0003] Before a neural network can analyze text, it needs to represent the text as word vectors that the neural network can understand. The ability of word vectors to extract and generalize text features, i.e., word embeddings, directly affects the accuracy and speed of downstream tasks. The concept of word embeddings was first introduced by Xu, Rudnicky, and Bengio in Neural Network Language Models (NNLM), which suggests that words in similar contexts are more likely to have similar meanings.

[0004] With the development of word embedding, word vector representation has begun to rely less on neural networks and embedding layers, and more on word context matrices to obtain word vector representations. In 2013, Mikolov proposed the Word2Vector model, which includes two frameworks: Skip-Gram and Continuous Bag-of-Words (CBOW). These two frameworks are very popular in NLP (Natural Language Processing) tasks and can be viewed as a two-layer neural network language model. In 2014, Pennington et al. proposed the GloVe model. Context matrix-based word embeddings are dense, distributed, and fixed-length word vectors, constructed using word co-occurrence statistics based on the distributed assumption. However, context matrix-based word embeddings cannot capture the semantic features of long sentences and texts containing words. Until 2018, the Transformer model emerged, reshaping many downstream NLP tasks. The BERT model is a bidirectional encoder based on the Transformer model. Unlike all previous encoding models, the BERT model is a context-based embedding model, while other popular embedding models such as word2vec are context-free.

[0005] Therefore, BERT is now the preferred model for word vector encoding of most large datasets. However, due to its masking mechanism, the word vector space encoded by BERT exhibits anisotropy, affecting feature extraction from social media texts and thus impacting downstream tasks related to public opinion analysis.

[0006] Specifically, Timothee Mickus et al., in their study of BERT's word vector space, found that while BERT exhibits a consistent trend, it doesn't fully meet the natural expectation of a semantic vector space—that similar words should be represented by similar vectors. Sara Rajaee et al. conducted a detailed study on the degradation problem in the BERT model, finding that in an anisotropic embedding space, word vectors are distributed within a narrow cone, and even unrelated words are considered to have high cosine similarity. This undesirable property hinders the representativeness of the embedding space and limits the diversity of encoded knowledge. To improve the contextual representation word vector degradation problem in BERT, some have proposed a method based on local clustering, using PCA (Principal Component Analysis) to cluster the embeddings and find principal components (PCs). The principal components represent the dominant direction of each cluster, and clearing the PCs makes the vector space more isotropic. However, clearing the PCs may destroy the semantic representation of the word vectors. Based on the idea of ​​clearing principal components (PCs), Yuxin Liang proposed using a set of learnable weights to remove several dominant directions in BERT embeddings. By adjusting word similarity, this indirectly adjusts the geometric characteristics of word vector distribution in the vector space, making it more isotropic. However, existing PCA methods, after clustering, result in a decrease in the representational meaning of word vectors. Moreover, social media texts are often short and colloquial, making the characteristic of low-frequency words being squeezed into a narrower cone and pushed away from the origin due to the masking mechanism even more pronounced. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides a text monitoring method and apparatus based on hybrid word vector encoding.

[0008] The technical problem to be solved by this invention is achieved through the following technical solution:

[0009] A text detection method based on hybrid word vector encoding includes:

[0010] Crawling public opinion texts;

[0011] The BERT model is used to encode the public opinion text to obtain the character vector corresponding to each character in the public opinion text;

[0012] The aforementioned public opinion text was segmented into multiple words;

[0013] From a pre-defined dictionary, a first word vector corresponding to each word is retrieved; wherein, the first word vector is a spatially anisotropic improved word vector obtained by clustering the initial word vectors corresponding to all words in the dictionary according to categories using linear discriminant analysis; the initial word vector is a word vector obtained by encoding the words in the dictionary using the WOBERT model;

[0014] Based on each of the aforementioned word vectors, a second word vector corresponding to each word in the public opinion text is obtained, and the first and second word vectors corresponding to each word in the public opinion text are merged to obtain an improved word vector corresponding to that word;

[0015] Public opinion monitoring is performed based on the improved word vectors corresponding to the words in the aforementioned public opinion text.

[0016] Optionally, the method further includes:

[0017] After crawling public opinion texts, depending on the data source of the texts, a relational database or a distributed database is selected to store the texts, so that the texts can be retrieved from the databases and encoded and segmented.

[0018] Optionally, the method further includes: after crawling the public opinion text, preprocessing the public opinion text, so as to encode and segment the preprocessed public opinion text; the preprocessing includes at least: text cleaning.

[0019] Optionally, after segmenting the public opinion text, the method further includes:

[0020] Detect new words appearing in the aforementioned public opinion text;

[0021] The sentiment attribute of the new word is determined based on its context, and the sentiment attribute is used as the category of the new word.

[0022] The new words and their categories are updated in the dictionary, and the initial word vectors corresponding to all words in the dictionary are clustered according to their categories using linear discriminant analysis.

[0023] Optionally, detecting new words appearing in the public opinion text includes:

[0024] New words were detected in a predetermined proportion of all public opinion texts using a new word detection tool.

[0025] Optionally, the public opinion monitoring includes: media report analysis, public opinion time-based heat analysis, public opinion regional heat analysis, and / or public opinion sentiment trend analysis.

[0026] Optionally, the method further includes: displaying the content of public opinion monitoring, and / or issuing public opinion warnings based on the content of public opinion monitoring, and / or processing public opinion based on the content of public opinion monitoring.

[0027] The present invention also provides a text monitoring device based on hybrid word vector encoding, comprising:

[0028] The text crawling module is used to crawl public opinion texts;

[0029] The encoding module is used to encode the public opinion text using the BERT model to obtain the character vector corresponding to each character in the public opinion text;

[0030] The word segmentation module is used to segment the public opinion text into multiple words;

[0031] The search module is used to search for the first word vector corresponding to each word in a preset dictionary; wherein, the first word vector is a spatially anisotropic word vector obtained by clustering the initial word vectors corresponding to all words in the dictionary according to categories using linear discriminant analysis; the initial word vector is a word vector obtained by encoding the words in the dictionary using the WOBERT model;

[0032] The vector merging module is used to obtain the second word vector corresponding to each word in the public opinion text based on each of the word vectors, and to merge the first word vector and the second word vector corresponding to each word in the public opinion text to obtain the improved word vector corresponding to the word.

[0033] The public opinion monitoring module is used to monitor public opinion based on the improved word vectors corresponding to the words in the public opinion text.

[0034] Optionally, the device further includes:

[0035] The storage module is used to select a relational database or a distributed database to store the public opinion text after the crawling module crawls it, based on the data source of the text. This allows for subsequent retrieval of the text from the database and its encoding and word segmentation.

[0036] Optionally, the device further includes: a new word detection module and a dictionary update module;

[0037] The new word detection module is used to detect new words appearing in the public opinion text; determine the sentiment attribute of the new word based on its context, and use the sentiment attribute as the category of the new word;

[0038] The dictionary update module is used to update the new words and their categories into the dictionary, and to re-use linear discriminant analysis to cluster the initial word vectors corresponding to all words in the dictionary according to their categories.

[0039] The text monitoring method based on hybrid word vector encoding provided by this invention segments the public opinion text into words. It then retrieves the first word vector corresponding to each word from a pre-defined dictionary. This first word vector is obtained by clustering the initial word vectors corresponding to all words in the dictionary according to categories using linear discriminant analysis, resulting in spatially anisotropic word vectors. This enhances the classification distinction of word vectors, making the spatial distance between word vectors of the same category closer, thus improving the problems of insufficient cosine value discrimination and anisotropic spatial features in the original BERT word vectors. Simultaneously, this invention uses the BERT model to encode the public opinion text, effectively preserving the contextual semantic features. Therefore, the improved word vector obtained by merging the second word vector (composed of word vectors encoded by the BERT model) with the spatially anisotropic improved first word vector retains the representational meaning of the BERT word vectors while improving the crowded vector space. Consequently, when conducting public opinion monitoring based on the improved word vectors, it can better classify and semantically recognize different public opinions, improving the monitoring effect.

[0040] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0041] Figure 1 This is a flowchart of a text monitoring method based on hybrid word vector encoding provided in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of the dictionary as preset in the embodiments of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the improvement of the initial word vector using nonlinear discriminant analysis in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram illustrating the use of a traffic splitting mechanism to store public opinion text in an embodiment of the present invention;

[0045] Figure 5 This is an illustrative public opinion monitoring process shown in an embodiment of the present invention;

[0046] Figure 6 The diagram shows the dimensionality reduction effect of the initial word vectors before improvement, obtained by the PCA algorithm.

[0047] Figure 7 The diagram shows the dimensionality reduction effect of the improved word vectors obtained by the PCA algorithm. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0049] The BERT model suffers from anisotropy in its encoding word vector space. Social media language contains a large number of colloquial and internet slang terms. These low-frequency and uncommon words cause the BERT encoding space to exhibit a narrow cone shape. Specifically, the similarity of words of the same category and the similarity of words of different categories are at the same level, which leads to a decrease in the accuracy and performance of downstream tasks in public opinion analysis.

[0050] To address the aforementioned problems, embodiments of the present invention provide a text monitoring method based on hybrid word vector encoding, such as... Figure 1 As shown, the method includes the following steps:

[0051] S10. Crawling public opinion texts.

[0052] Specifically, event data can be tracked from multiple sources of public opinion, such as source a, source b, and source c. Among them, source a mainly involves text-based social media platforms, which have a high degree of autonomy in posting, a relatively lenient review mechanism, high participation from young people, and are prone to the fermentation of public opinion.

[0053] In step S10, the crawling of public opinion text is mainly achieved through the web interface. The crawled content can be crawled according to keywords, continuously obtaining one or more keyword search results, and searching for text in the crawled text that contains the specified keywords. At the same time, in order to ensure the timeliness and accuracy of the crawled content, the search time range can be specified.

[0054] Optionally, after crawling public opinion texts, the public opinion texts can be preprocessed before encoding and word segmentation. The preprocessing mentioned here includes at least text cleaning, but is not limited to this.

[0055] S20. Use the BERT model to encode the public opinion text and obtain the character vector corresponding to each character in the public opinion text.

[0056] Specifically, the public opinion text is input into the BERT model, which directly encodes the text and outputs the word vector corresponding to each character. Here, using the BERT model to encode the public opinion text retains the advantages of BERT model encoding: complete semantic expression and good preservation of the semantic features of the context.

[0057] S30. Segment the public opinion text to obtain multiple words.

[0058] Here, the JIEBA algorithm can be used to segment public opinion texts. The JIEBA algorithm is a Chinese word segmentation algorithm based on a statistical dictionary and a hidden Markov model. It achieves word segmentation and annotation by building a dictionary and a probability model, scanning and statistically analyzing the text. JIEBA word segmentation is efficient, accurate, and flexible, and is widely used in Chinese natural language processing.

[0059] S40. From the preset dictionary, find the first word vector corresponding to each word; wherein, the first word vector is a spatially anisotropic improved word vector obtained by clustering the initial word vectors corresponding to all words in the dictionary according to categories using linear discriminant analysis; the initial word vector is a word vector obtained by encoding the words in the dictionary using the WOBERT model.

[0060] Understandably, the dictionary contains a large number of words, each with a corresponding category and a first word vector. These first word vectors are obtained by improving the word vectors using linear discriminant analysis. Specifically, the WOBERT model is used to encode the words in the dictionary, obtaining an initial word vector for each word; then, linear discriminant analysis is used to cluster all the initial word vectors according to the categories of the words they correspond to, obtaining the first word vector for each word in the dictionary.

[0061] In this embodiment of the invention, the dictionary can be configured as follows: Figure 2 The structure shown is used for classification and storage. Each classification lexicon is created as a CSV file, similar to Excel, with three columns containing the word, part-of-speech tag, and first word vector. The part-of-speech tag refers to the identifier of each classified word; for example, nouns are tagged as 1, verbs as 2, classifiers as 3, and so on.

[0062] Linear Discriminant Analysis (LDA) is a classic pattern recognition and machine learning method used to find the most distinguishing orientations between different classes in a feature space. It is commonly used for dimensionality reduction and classification tasks. The goal of LDA, given class labels, is to find a projection orientation that makes samples of the same class as close as possible and samples of different classes as far apart as possible. This projection orientation is determined by maximizing inter-class distance and minimizing intra-class distance.

[0063] In practice, the steps to improve the initial word vectors using linear discriminant analysis are as follows: calculate the mean vector for each class; calculate the within-class scatter matrix; calculate the between-class scatter matrix; and solve a generalized eigenvalue problem to obtain the optimal projection direction (linear discriminant vector). The resulting word vector with the projection direction is the dimensionality-reduced word vector, i.e., the first word vector, which is essentially a subspace of the BERT word vectors. This part of the word vectors enhances the classification distinction of the BERT word vectors, and the spatial distance between word vectors of the same class is closer. This improves the problems of insufficient discriminative power of the cosine value and isotropic spatial features in the original BERT word vectors.

[0064] Furthermore, after processing the initial word vectors through linear discriminant analysis, the dimensionality of the processed word vectors can be increased by padding with zeros (e.g., ...). Figure 3 (as shown in the image) so that subsequent word vector merging operations can be performed.

[0065] S50. Based on each word vector, obtain the second word vector corresponding to each word in the public opinion text, and merge the first word vector and the second word vector corresponding to each word in the public opinion text to obtain the improved word vector corresponding to that word.

[0066] Specifically, for each word in the public opinion text, the character vectors corresponding to its contained characters are merged into a second word vector. Then, this second word vector is merged with the first word vector corresponding to the word to obtain the improved word vector corresponding to the word.

[0067] In this embodiment of the invention, in order to address the problem of isotropic coding space of BERT coding model, the idea of ​​using linear discriminant analysis method is proposed to improve the problem of large distance between BERT word vectors of the same category and the anisotropic space. The improved word vector includes two parts: the second word vector based on the original coding of BERT model and the improved first word vector. Therefore, the improved word vector retains the representational meaning of BERT word vector and improves the crowded vector space.

[0068] S60. Conduct public opinion monitoring based on improved word vectors corresponding to words in public opinion texts.

[0069] Specifically, based on improved word vectors corresponding to words in public opinion texts, text analysis algorithms are used for public opinion monitoring. These algorithms include word frequency analysis, semantic analysis, topic analysis, sentiment analysis, entity recognition, relationship analysis, time series analysis, text classification, and text clustering.

[0070] For example, the aforementioned public opinion monitoring may include: media report analysis, public opinion time-based heat analysis, public opinion regional heat analysis, and / or public opinion sentiment trend analysis.

[0071] Figure 5 The diagram illustrates, for example, a process diagram for public opinion monitoring based on improved word vectors corresponding to words in public opinion text.

[0072] Specifically, for Category A public opinion data, downstream text analysis algorithms can be used to obtain a Category A public opinion word cloud by analyzing keywords and the frequency of related hot words; the time and location of text publication can be extracted to obtain a line graph of the time heat change and a regional heat map of Category A public opinion; and the sentiment trend of Category A can be analyzed to obtain the sentiment attitude towards this event.

[0073] A public opinion heatmap can be generated through the following process: First, convert the crawled timestamps into a usable time format (e.g., hour, day, week) and normalize the time. Second, count the quantity of relevant content by time unit (e.g., hour, day) to calculate the heat data within different time units. Third, use visualization tools to represent the calculated public opinion heat data using line charts or bar charts to illustrate the relationship between time and public opinion heat. Examples of visualization tools include Python-based data visualization tools like Tableau and Power BI.

[0074] A regional heat map of public opinion can be generated as follows: Count the amount of relevant content by region to generate a heat value for each region. Then, use the visualization tools described above to display the public opinion heat in different regions.

[0075] For reports from source c, the sentiment trend of the data in category c can be analyzed to determine the popularity of the reports from source c based on the number and frequency of media coverage. This sentiment trend analysis can be achieved by feeding improved word vectors into an LSTM (Long Short-Term Memory) network for sentiment analysis.

[0076] For the content of the report from source b, the sentiment analysis method can be the same as that used for the report from source c, and it can also be combined with manual analysis in the background. In addition, since the content of the report from source b is generally less and more important, the entire text of the public opinion can be crawled and displayed on the front-end page.

[0077] In summary, the text monitoring method based on hybrid word vector encoding provided in this invention segmentes the public opinion text and retrieves the first word vector corresponding to each word from a pre-defined dictionary. This first word vector is a spatially anisotropic word vector obtained by clustering the initial word vectors corresponding to all words in the dictionary according to categories using linear discriminant analysis. This enhances the classification distinction of the word vectors, making the spatial distance between word vectors of the same category closer, thus improving the problems of insufficient cosine value discrimination and anisotropic spatial features of the original BERT word vectors. Simultaneously, this invention utilizes the BERT model to encode the public opinion text, effectively preserving the contextual semantic features. Therefore, the improved word vector obtained by merging the second word vector (composed of word vectors encoded by the BERT model) with the spatially anisotropic improved first word vector retains the representational meaning of the BERT word vectors while improving the crowded vector space. Consequently, when conducting public opinion monitoring based on the improved word vector, it can better classify and semantically recognize different public opinions, improving the monitoring effect.

[0078] Optionally, in one implementation, the text monitoring method with hybrid word vector encoding provided in this embodiment of the invention may further include: after crawling public opinion text, selecting a relational database or a distributed database to store the public opinion text according to the data source of the public opinion text, so as to subsequently obtain the public opinion text from the database and encode and segment it.

[0079] For details, see Figure 4 The responses from government and enterprise websites are stored using traditional databases (Category B), while Category A public opinion content is stored using a distributed database. Category C data is categorized into small and large events based on the scope and volume of the event. Small events typically have a smaller data volume and can be stored in a relational database. Events that significantly impact database read speed (e.g., a 4GB data limit can be set) are considered large events and are then stored in a distributed database.

[0080] Understandably, the evaluation and processing of Category B data often involves smaller amounts of text, more client-side calls, and higher security and confidentiality requirements. These scenarios demand structured data, high security, and reliability. Public opinion analysis, in particular, involves more frequent calls and more detailed research, making relational databases a better choice due to their fast read speed, high call frequency, and smaller text volume. However, for scenarios requiring the processing of large amounts of unstructured data such as social media language, high concurrency access, and high performance, non-relational databases, i.e., distributed databases, are more suitable.

[0081] Therefore, in this embodiment of the invention, the captured data is stored in different databases according to different data sources. The splitting mode is used to combine traditional relational databases with combined distributed databases, which helps to improve the flexible use of different public opinion texts in the future.

[0082] Optionally, in one implementation, the text monitoring method using hybrid word vector encoding provided in this embodiment of the invention may further include: detecting new words appearing in public opinion text; determining the sentiment attribute of the new word based on its context, and using the sentiment attribute as the category of the new word; updating the new word and its category to the dictionary, and re-clustering the initial word vectors corresponding to all words in the dictionary according to their categories using linear discriminant analysis. For example, in Figure 5 In this study, 10% to 20% of public opinion texts were randomly selected for new word detection and their emotional attributes were identified. These texts were then included in the dictionary.

[0083] New word detection can be achieved using new word detection tools, such as NLPIR developed by the Chinese Academy of Sciences, but is not limited to this.

[0084] It is understandable that the main user base of social networks (source a) is young people, who are adept at using online language to express their opinions on current events. The emergence of new online buzzwords is often time-sensitive, and their semantics may differ significantly from their literal meaning. The emotional meaning of new buzzwords generated by a single event needs to be judged based on the semantics of the text; judging solely by the literal meaning increases the probability of misinterpretation.

[0085] To address the aforementioned problems, this invention determines the sentiment attribute of words based on their context. Specifically, it analyzes the usage of words in specific contexts to determine their sentiment tendency, i.e., positive, negative, or neutral. This method has wide applications in natural language processing, text sentiment classification, and other fields. In natural language processing, corpus-based sentiment assessment is a common method. By mining information from large-scale corpora and analyzing the grammatical characteristics and usage contexts of words, their sentiment attributes can be determined.

[0086] In addition, when detecting new words in public opinion texts, new word detection tools can be used to detect new words in all public opinion texts; or, a preset proportion of public opinion texts can be selected for new word detection, such as 10% to 20% of all public opinion texts.

[0087] Understandably, when a novel, contextualized word with strong emotional connotations appears in a relevant event, it often exhibits high frequency and strong emotional expression within the sentence. Traditional new word identification methods, however, rely on meticulous, comprehensive analysis, which is resource-intensive. Existing new word recognition algorithms primarily depend on statistical methods such as word frequency statistics, requiring analysis of large amounts of text content. This results in low efficiency and insufficient professionalism and accuracy for identifying new words specific to particular events. Furthermore, they cannot quickly analyze the attributes of new words, such as emotional connotations and parts of speech, for downstream tasks, thus hindering rapid text analysis. In contrast, this invention focuses solely on detecting new words in public opinion texts, randomly selecting a certain proportion of these texts for new word detection. This allows for more efficient and targeted detection of new words relevant to downstream text analysis and processing.

[0088] In this embodiment of the invention, considering the frequent appearance of new words in the real network environment, the dictionary is updated by detecting new words in public opinion texts and judging new words in different tasks in real time. This facilitates the downstream tasks of public opinion analysis and enhances the accuracy of text processing tasks such as text sentiment judgment.

[0089] Optionally, in one implementation, the text monitoring method using hybrid word vector encoding provided in this embodiment of the invention may further include: displaying the content of public opinion monitoring, and / or issuing public opinion warnings based on the content of public opinion monitoring, and / or processing public opinion based on the content of public opinion monitoring.

[0090] For details, see Figure 5 This system can display public opinion time-based heatmaps, regional heatmaps, A-type data sentiment maps, source b reports, and the handling measures described in source b reports on the front end. Furthermore, based on the analysis of source c and source b reports, the system can combine source c and A-type public opinion heatmaps to determine the scope and spread of the event, thereby deciding whether to issue a public opinion warning. If no warning is issued, monitoring continues. If a warning is issued, relevant governance opinions can be provided based on source c and source b reports, combined with relevant laws and regulations, i.e., public opinion management. Here, the provision of governance opinions can be achieved by identifying content related to the "suggestions" through topic modeling (such as LDA). Topic modeling can be implemented using a topic model based on Latent Dirichlet Allocation (LDA).

[0091] In a specific example, based on vocab.txt in the BERT model, a dictionary of thousands of words is compiled, with both positive and negative part-of-speech categories. The BERT model is used to encode the words in the dictionary and integrate them into initial word vectors, which are then saved to the dictionary. The Wordpiece algorithm is then used to segment individual sentences, and the words are compiled into the dictionary. These words are encoded and saved as an indexed dictionary. Linear discriminant analysis is used to achieve two objectives: minimizing the within-class scatter matrix and maximizing the between-class scatter matrix, thereby improving the initial word vectors. Figure 6 The image shows the dimensionality reduction effect of the initial word vectors before improvement, obtained through the PCA algorithm. Figure 7 The diagram shows the dimensionality reduction effect of the improved word vectors obtained by the PCA algorithm. The ordinates and abscissas refer to the projections of the multidimensional vectors onto the two-dimensional plane. Figure 7 The dots of the same color inside refer to words expressing the same type of emotion.

[0092] The spatial isotropy of word vectors is specifically manifested in the fact that the spatial distance between words of the same category and the distance between words of different categories is not significantly different, and it lacks the ability to distinguish word classes. For example Figure 6 As shown, the original word vectors encoded by the BERT model have poor classification performance, with almost no classification ability and a classification accuracy of less than 50%. However, the improved word vectors show a significant improvement in classification performance, such as... Figure 7 As shown, improved word vectors of the same type are more similar in the word vector encoding space (the same color clusters more), and the accuracy eventually settles at around 83%. This indicates that the new hybrid word vectors obtained based on the idea of ​​linear discriminant analysis greatly improve the problem of spatial anisotropy of the initial word vectors.

[0093] The anisotropy of the encoding space manifests as the distances between words of the same type in the encoding vector space not conforming to actual meaning; words with similar meanings are not necessarily closer together. Figure 7 As can be seen, the improved word vectors have closer distances to words of the same category, effectively improving spatial anisotropy and improving the word vector encoding space while ensuring that the meaning of the word vectors does not change significantly. At the same time, the improved word vectors contain a larger proportion of the initial word vectors, greatly preserving the original features of the BERT word vectors. This effectively avoids the problem of decreased meaning of word vectors caused by removing principal components after clustering in existing PCA methods.

[0094] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0095] Based on the same inventive concept, this invention also provides a text monitoring device based on hybrid word vector encoding, including: a text crawling module, an encoding module, a word segmentation module, a search module, a vector merging module, and a public opinion monitoring module.

[0096] The text crawling module is used to crawl public opinion texts.

[0097] The encoding module is used to encode public opinion text using the BERT model to obtain the character vector corresponding to each character in the public opinion text;

[0098] The word segmentation module is used to segment public opinion texts into multiple words;

[0099] The search module is used to find the first word vector corresponding to each of the above words from the preset dictionary; wherein, the first word vector is a spatially anisotropic word vector obtained by clustering the initial word vectors corresponding to all words in the dictionary according to categories using linear discriminant analysis; the initial word vector is a word vector obtained by encoding the words in the dictionary using the WOBERT model;

[0100] The vector merging module is used to obtain the second word vector corresponding to each word in the public opinion text based on the individual word vectors, and to merge the first word vector and the second word vector corresponding to each word in the public opinion text to obtain the improved word vector corresponding to that word.

[0101] The public opinion monitoring module is used to monitor public opinion based on the improved word vectors corresponding to words in the public opinion text.

[0102] Optionally, the above-mentioned device further includes:

[0103] The storage module is used to select a relational database or a distributed database to store the public opinion text after the crawling module crawls it, based on the data source of the text. This allows for subsequent retrieval of the text from the database and its encoding and word segmentation.

[0104] Optionally, the above-mentioned device further includes: a new word detection module and a dictionary update module;

[0105] The new word detection module is used to detect new words appearing in public opinion texts; it determines the sentiment attribute of new words based on their context and uses this sentiment attribute as the category of the new words;

[0106] The dictionary update module is used to update the dictionary with new words and their categories, and to re-use linear discriminant analysis to cluster the initial word vectors corresponding to all words in the dictionary according to their categories.

[0107] The text monitoring device based on hybrid word vector encoding provided in this invention can be a software system. The system's overall architecture is distributed, divided into five layers based on the different functionalities of each part: a data acquisition layer, a data storage layer, a text encoding layer, a business layer, and a presentation layer. The presentation layer primarily displays the system's functions through front-end page design. It mainly includes two parts: the front-end display of the business operations performed by the business layer, and the event processing, early warning results, and processing content.

[0108] As the device embodiment is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the description of the method embodiment. The embodiments of the present invention will not be described again.

[0109] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0110] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0111] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the description of this invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0112] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as "modules" or "systems." Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The computer program may be stored / distributed in a suitable medium, provided with or as part of other hardware, or may take other distribution forms, such as via the Internet or other wired or wireless telecommunications systems.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A text detection method based on hybrid word vector encoding, characterized in that, include: Crawling public opinion texts; The BERT model is used to encode the public opinion text to obtain the character vector corresponding to each character in the public opinion text; The aforementioned public opinion text was segmented into multiple words; From a pre-defined dictionary, a first word vector corresponding to each word is retrieved; wherein each word in the dictionary has a corresponding category and a first word vector; the first word vector is a spatially anisotropic word vector obtained by clustering the initial word vectors corresponding to all words in the dictionary according to categories using linear discriminant analysis, which is essentially a subspace of BERT word vectors, used to enhance the classification distinction of the initial word vectors and make the spatial distance between initial word vectors of the same category closer; the initial word vectors are word vectors obtained by encoding the words in the dictionary using the WOBERT model; Based on each of the aforementioned word vectors, a second word vector corresponding to each word in the public opinion text is obtained, and the first and second word vectors corresponding to each word in the public opinion text are merged to obtain an improved word vector corresponding to that word; Public opinion monitoring is performed based on the improved word vectors corresponding to the words in the aforementioned public opinion text; After segmenting the public opinion text, the method further includes: Detect new words appearing in the aforementioned public opinion text; The sentiment attribute of the new word is determined based on its context, and the sentiment attribute is used as the category of the new word. The new words and their categories are updated in the dictionary, and the initial word vectors corresponding to all words in the dictionary are clustered according to their categories using linear discriminant analysis.

2. The text monitoring method based on hybrid word vector encoding according to claim 1, characterized in that, The method further includes: After crawling public opinion texts, depending on the data source of the texts, a relational database or a distributed database is selected to store the texts, so that the texts can be retrieved from the databases and encoded and segmented.

3. The text monitoring method based on hybrid word vector encoding according to claim 1, characterized in that, The method further includes: after crawling public opinion text, preprocessing the public opinion text, and then encoding and segmenting the preprocessed public opinion text; the preprocessing includes at least: text cleaning.

4. The text monitoring method based on hybrid word vector encoding according to claim 1, characterized in that, Detecting new words appearing in the aforementioned public opinion text includes: New words were detected in a predetermined proportion of all public opinion texts using a new word detection tool.

5. The text detection method based on hybrid word vector encoding according to claim 1, characterized in that, The public opinion monitoring includes: media report analysis, public opinion time-based heat analysis, public opinion regional heat analysis, and / or public opinion sentiment trend analysis.

6. The text detection method based on hybrid word vector encoding according to claim 5, characterized in that, The method further includes: displaying the content of public opinion monitoring, and / or issuing public opinion warnings based on the content of public opinion monitoring, and / or processing public opinion based on the content of public opinion monitoring.

7. A text monitoring device based on hybrid word vector encoding, characterized in that, include: The text crawling module is used to crawl public opinion texts; The encoding module is used to encode the public opinion text using the BERT model to obtain the character vector corresponding to each character in the public opinion text; The word segmentation module is used to segment the public opinion text into multiple words; The search module is used to search for the first word vector corresponding to each word in a preset dictionary; wherein each word in the dictionary has a corresponding category and a first word vector; the first word vector is a spatially anisotropic word vector obtained by clustering the initial word vectors corresponding to all words in the dictionary according to categories using linear discriminant analysis, which is essentially a subspace of BERT word vectors, used to enhance the classification distinction of the initial word vectors and make the spatial distance between initial word vectors of the same category closer; the initial word vector is a word vector obtained by encoding the words in the dictionary using the WOBERT model; The vector merging module is used to obtain the second word vector corresponding to each word in the public opinion text based on each of the word vectors, and to merge the first word vector and the second word vector corresponding to each word in the public opinion text to obtain the improved word vector corresponding to the word. The public opinion monitoring module is used to monitor public opinion based on the improved word vectors corresponding to the words in the public opinion text; The device also includes: a new word detection module and a dictionary update module; The new word detection module is used to detect new words appearing in the public opinion text; determine the sentiment attribute of the new word based on its context, and use the sentiment attribute as the category of the new word; The dictionary update module is used to update the new words and their categories into the dictionary, and to re-use linear discriminant analysis to cluster the initial word vectors corresponding to all words in the dictionary according to their categories.

8. The text monitoring device based on hybrid word vector encoding according to claim 7, characterized in that, The device further includes: The storage module is used to select a relational database or a distributed database to store the public opinion text after the crawling module crawls it, based on the data source of the text. This allows for subsequent retrieval of the text from the database and its encoding and word segmentation.

Images