Method, device and equipment for network public opinion text similarity calculation and storage medium

Abstract

The invention relates to the technical field of deep learning, and discloses a network public opinion similarity calculation method, device and equipment and a storage medium, and the method comprises the steps: obtaining network public opinion text data, and training a constructed similarity calculation model based on the network public opinion text data; performing text marking processing on the network public opinion text data to obtain a text marking processing result, performing random mask operation on the text marking processing result, and based on a mask position of a mask text, obtaining a prediction word and a prediction probability that the mask position is the prediction word; constructing a loss function based on the difference between the prediction probability and the text marking processing result, and obtaining a trained similarity calculation model under the condition that the loss function converges; and based on the trained similarity calculation model, obtaining a similarity calculation result corresponding to the to-be-tested network public opinion text data. And the requirement of public opinion text similarity accuracy is met.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method, device, equipment and storage medium for calculating the similarity of online public opinion. Background Art

[0002] With the popularity of online ride-hailing services, the public's attention to online ride-hailing related news and opinions has gradually increased. In order to better understand the public's views and emotions on online ride-hailing services, online ride-hailing companies need to effectively analyze and process a large amount of public opinion news. The online ride-hailing network public opinion news processing system in related technologies mostly uses traditional text processing and feature extraction technology, which has the problem of repeated appearance of the same articles and cannot meet the accuracy requirements. Summary of the invention

[0003] In view of this, the present invention provides a method, device, equipment, storage medium and program product for calculating the similarity of network public opinion text, so as to solve the problem that the online car-hailing network public opinion news processing system in the related technology mostly adopts traditional text processing and feature extraction technology, and the same articles appear repeatedly and cannot meet the accuracy requirements.

[0004] In a first aspect, the present invention provides a method for calculating the similarity of online public opinion texts, the method comprising: obtaining online public opinion text data, and constructing a similarity calculation model based on the training of the online public opinion text data; performing text tokenization processing on the online public opinion text data to obtain a text tokenization processing result, performing a random masking operation on the text tokenization processing result, and obtaining a predicted word and a predicted probability that the mask position is the predicted word based on a mask position of the masked text; constructing a loss function based on the difference between the predicted probability and the text tokenization processing result, and obtaining a trained similarity calculation model when the loss function converges; and obtaining a similarity calculation result corresponding to the online public opinion text data to be tested based on the trained similarity calculation model.

[0005] In an optional implementation, constructing a loss function based on the difference between the predicted probability and the text tokenization processing result includes:

[0006] The loss function is constructed by calculating the cross entropy loss between the predicted probability and the text tokenization result. The cross entropy loss H(y, y) ′ ) is calculated by the following formula:

[0007]

[0008] Among them, y iis the one-hot encoding of the i-th component of the real data in the text tokenization processing result. If the real data is of the i-th category, then y i =1, otherwise, y i =0;y i ′ is the predicted probability of the i-th category.

[0009] In an optional embodiment, a random masking operation is performed on the text tokenization processing result, and based on the mask position of the masked text, a predicted word and a predicted probability of the mask position being the predicted word are obtained, including: randomly selecting some words in the text tokenization processing result, and replacing the some words with the masked text; generating a predicted word probability distribution at the mask position, the predicted word probability distribution including a plurality of predicted words and a target predicted word, and the target predicted probability characterizing the possibility of the target predicted word at the mask position.

[0010] In an optional implementation, the network public opinion text data is subjected to text tokenization processing to obtain a text tokenization processing result, including: performing word segmentation on the Chinese text in the network public opinion text data based on a word segmentation tool, and performing word segmentation on the English text in the network public opinion text data based on spaces and / or punctuation marks to obtain a word segmentation result; marking the parts of speech of words and / or phrases in the word segmentation result; performing named entity recognition based on the word segmentation result; and obtaining the text tokenization processing result, wherein the text tokenization processing result includes the text, part of speech and named entity tag corresponding to the word, and the text, part of speech and named entity tag corresponding to the phrase.

[0011] In an optional embodiment, based on the trained similarity calculation model, a similarity calculation result corresponding to the network public opinion text data to be tested is obtained, including: inputting the first network public opinion text data and the second network public opinion text data into the trained similarity calculation model to obtain a first text vector and a second text vector; calculating the similarity between the first text vector and the second text vector to obtain the similarity calculation result.

[0012] In an optional implementation, the aforementioned method for calculating the similarity of online public opinion texts further includes: grouping the online public opinion text data based on the similarity calculation result, and the similarity calculation result greater than a first preset threshold is a first group.

[0013] In a second aspect, the present invention provides a device for calculating the similarity of online public opinion texts, the device comprising: a model construction module, used to obtain online public opinion text data, and a similarity calculation model constructed based on the online public opinion text data training; a model training module, used to perform text tokenization processing on the online public opinion text data to obtain a text tokenization processing result, perform a random masking operation on the text tokenization processing result, and obtain a predicted word and a predicted probability that the mask position is the predicted word based on a mask position of the masked text; a loss function is constructed based on the difference between the predicted probability and the text tokenization processing result, and a trained similarity calculation model is obtained when the loss function converges; a similarity calculation module, used to obtain a similarity calculation result corresponding to the online public opinion text data to be tested based on the trained similarity calculation model.

[0014] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for calculating the similarity of online public opinion text according to the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for calculating the similarity of online public opinion texts according to the first aspect or any corresponding embodiment thereof.

[0016] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the method for calculating the similarity of online public opinion texts according to the above-mentioned first aspect or any corresponding embodiment thereof.

[0017] Based on the online public opinion text data, the similarity calculation model is trained and evaluated so that the similarity calculation model can calculate the similarity between texts. Through text tokenization, the original text can be converted into a format that is easier for machine processing. Through random mask operations and predictions, the similarity calculation model can learn the contextual representation of the text. When the loss function converges, the accuracy of the similarity calculation model can be improved. In this way, the similarity between the online public opinion text data can be accurately calculated, meeting the requirements for the accuracy of the online public opinion text similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a process for calculating the similarity of network public opinion texts provided by an embodiment of the present invention is shown;

[0020] Figure 2 A schematic diagram showing the relationship between the number of iterations and the loss of the cross entropy loss function in an embodiment of the present invention is shown;

[0021] Figure 3 A schematic diagram of a random mask operation process in an embodiment of the present invention is shown;

[0022] Figure 4 The invention shows a device for calculating the similarity of network public opinion texts provided by an embodiment of the invention;

[0023] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] The methods for calculating the similarity of online ride-hailing network public opinion news in related technologies are mainly based on artificial feature extraction and traditional machine learning algorithms, and use artificially designed features and classifiers to identify and classify news. For example, news keywords, topics, or sentiment features may be extracted, and classifiers such as support vector machines (SVM) or random forests may be used to identify news similarities. However, the efficiency and accuracy of this method are limited, and it cannot effectively process news texts with different contexts, different expressions, and different topics. The above method has the following three problems:

[0026] First, the processing efficiency is low. Due to the limitations of text processing and feature extraction technology in the above methods, the processing efficiency cannot meet the needs of online ride-hailing companies for public opinion management.

[0027] Secondly, the accuracy is poor. The online car-hailing network public opinion news similarity calculation system in the related technology is prone to errors when processing a large amount of news data, and cannot meet the accuracy requirements, resulting in the repetition of the same articles;

[0028] Thirdly, the system performance is unstable. Since the above method relies on artificially designed features, it is easily affected by context changes and noise, resulting in unstable system performance.

[0029] The embodiments of the present invention propose a method, apparatus, device, storage medium and computer program product for calculating the similarity of online public opinion texts, aiming to improve the efficiency, accuracy and system stability of news similarity calculation and provide online car-hailing companies with a more efficient and reliable public opinion management solution.

[0030] The concepts involved in the embodiments of the present invention are introduced below. Deep Learning is a technology that realizes advanced data representation and analysis by building a multi-layer neural network. Its core is to learn complex feature representations from a large amount of input data through nonlinear transformations of multiple neural network layers. It is a branch of machine learning.

[0031] The similarity calculation of online public opinion news uses deep learning technology to calculate the similarity of public opinion news related to online ride-hailing. Through natural language processing, text feature extraction and similarity evaluation technologies, the similarity measurement between news content can be achieved, providing online ride-hailing companies with functions such as public opinion monitoring, information management and service quality monitoring.

[0032] Natural Language Processing (NLP), which converts natural language text into a form that can be understood and processed by machines, is one of the important applications of deep learning in the field of language processing.

[0033] Text Feature Extraction can extract feature information from text, including word frequency, word order, grammatical structure, semantic information, etc.

[0034] Similarity Assessment can evaluate the similarity between two or more data points and can be applied to fields such as text analysis, recommendation systems, or image recognition.

[0035] The attention mechanism allows the neural network to focus on the important parts of the input data and improve the performance of the model in processing complex tasks by assigning different weights to different parts of the input data. The attention mechanism has a wide range of applications in fields such as natural language processing and image recognition.

[0036] According to an embodiment of the present invention, a method embodiment for calculating the similarity of online public opinion text is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] In this embodiment, a method for calculating the similarity of network public opinion text is provided, which can be used for a terminal with an operating system, such as a mobile phone, a tablet computer, a notebook, a server or a desktop computer, etc. Figure 1 A flow chart of a method for calculating the similarity of network public opinion texts provided by an embodiment of the present invention is shown. Figure 1 As shown, the process includes the following steps:

[0038] Step S101, obtaining network public opinion text data, and constructing a similarity calculation model based on network public opinion text data training.

[0039] In this step, the acquisition of online public opinion text data can be a large amount of real online ride-hailing online public opinion news text data randomly selected from an online database, or web page data obtained using crawler technology. Online public opinion text data can include: news texts with different contexts, different expressions, and different topics and their corresponding labels. The label can be whether an article is similar to another article. If similar, the label is 1, otherwise, the label is 0.

[0040] The similarity calculation model can be cosine similarity, Jaccard similarity, edit distance, or machine learning or deep learning model. For example, Bidirectional Encoder Representations from Transformers (BERT), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), these models are excellent in processing text and sequence data, especially in the task of calculating the similarity of online public opinion text. Or it can be a Generative Adversarial Networks (GANs) model, which can generate realistic text data through adversarial training, which is particularly important in the case of public opinion news generation and enhancement.

[0041] To obtain the website information that needs to be sorted, the crawler engine can be used to collect online public opinion text data, preprocess the data and perform feature engineering to generate training sets, test sets, and validation sets. Features can include word frequency, word order, grammatical structure, and semantic information of online public opinion text. Different text feature extraction methods can be used, such as TF-IDF, word embedding, etc. Word embedding can include Word2Vec or GloVe, etc., or a combination of multiple feature extraction methods can be used to improve the robustness of this solution.

[0042] Step S102, tokenize the network public opinion text data to obtain a text tokenization result, perform a random masking operation on the text tokenization result, and obtain the predicted word and the predicted probability of the masked position being the predicted word based on the mask position of the masked text; based on the difference between the predicted probability and the text tokenization result, construct a loss function, and when the loss function converges, obtain a trained similarity calculation model.

[0043] In this step, the online public opinion text data is tokenized to obtain text tokenization results, which may include word segmentation; adding special tags, such as tags for the beginning and end of sentences; converting words into numerical forms that can be understood by the similarity calculation model, such as word embedding indexes, etc.

[0044] Performing a random masking operation on the result of text tokenization can be to randomly select a certain proportion of words in the result of text tokenization and replace them with special mask tags, such as [mask] in BERT. Based on the mask position of the masked text, the predicted word and the predicted probability of the mask position being the predicted word are obtained. The masked text can be processed using a similarity calculation model. The similarity calculation model will try to predict the original word at the mask position based on the context and assign a prediction probability to each possible word. The prediction probability is used to characterize the possibility that the word is the original word at the mask position.

[0045] Based on the difference between the predicted probability and the result of text tokenization, a loss function is constructed, which can be a cross entropy loss function. By loading the training set and the test set, a data iterator is generated, and a mask operation of the expected proportion is randomly performed for the step-by-step calling of the iterative model. Initialize the similarity calculation model and related hyperparameters, including: input layer, multi-layer bidirectional transformer (transformer) encoder layer, output layer, loss function, number of training rounds, and the size of each training data.

[0046] Start training until the loss function converges. When the loss function converges, a trained similarity calculation model is obtained. The model parameters can be adjusted using optimization algorithms such as gradient descent to minimize the loss function, and the steps of training the similarity calculation model are repeated until the loss function converges, that is, the performance of the similarity calculation model is no longer significantly improved. After the loss function converges, a trained similarity calculation model is obtained. After the aforementioned training process, the similarity calculation model has learned the contextual representation of the text. In order to perform similarity calculation, a specific similarity calculation layer can be further added to the similarity calculation model or a specific similarity measurement method can be adopted, such as a cosine similarity calculation method.

[0047] Step S103, based on the trained similarity calculation model, obtain the similarity calculation result corresponding to the network public opinion text data to be tested.

[0048] In this step, the network public opinion text data to be tested can be preprocessed, including: word segmentation, removal of stop words or part-of-speech tagging, etc., to ensure that the text format is consistent with the format used when training the model. The preprocessed network public opinion text data to be tested is subjected to text tokenization processing, and the tokenization processing result of the test text is input into the trained similarity calculation model to obtain the similarity score or vector representation output by the similarity calculation model. Based on the similarity score or vector representation output by the model, the similarity between the network public opinion text data to be tested and the reference text is calculated. The reference text can be a text in a database, and the similarity calculation can be achieved by methods such as cosine similarity or Euclidean distance.

[0049] The method for calculating the similarity of online public opinion texts provided in this embodiment is based on online public opinion text data, trains and evaluates a similarity calculation model, so that the similarity calculation model can calculate the similarity between texts. Through text tokenization processing, the original text can be converted into a format that is easier for machine processing. Through random mask operations and predictions, the similarity calculation model is helped to learn the contextual representation of the text. When the loss function converges, the accuracy of the similarity calculation model can be improved. In this way, the similarity between online public opinion text data can be accurately calculated, meeting the requirements for the accuracy of online public opinion text similarity.

[0050] In some optional embodiments, based on the difference between the predicted probability and the text tokenization processing result, a loss function is constructed, including: by calculating the cross entropy loss between the predicted probability and the text tokenization processing result, a loss function is constructed, the cross entropy loss H(y, y ′ ) is calculated by the following formula:

[0051]

[0052] Among them, y i is the unique hot encoding of the i-th component of the real data in the text tokenization processing result. If the real data is of the i-th category, then y i =1, otherwise, y i =0;y i ′ is the predicted probability of the i-th category.

[0053] In this embodiment, in the training set of the similarity calculation model, each sample has a true label y and a predicted probability y of the similarity calculation model. ′ , for each sample, the cross entropy loss H(y,y ′ ) is calculated as shown above, where the logarithmic function log is usually based on the natural logarithm ln, but other bases can also be used. Figure 2 FIG. 4 is a schematic diagram showing the relationship between the number of iterations and the loss of the cross entropy loss function in an embodiment of the present invention. Figure 2 As shown in the figure, the horizontal axis represents the number of iterations and the vertical axis represents the loss (Loss).

[0054] Based on the cross entropy loss H(y,y ′ ) can measure the difference between the predicted distribution and the true distribution of the similarity calculation model in the problem of online public opinion text classification. Based on the concept of cross entropy in information theory, the difference between the two probability distributions can be quantified.

[0055] You can also use indicators such as precision, recall, F1 score, average accuracy or mean square error to evaluate the performance of the model. For similarity calculation models, response time, error rate or real-time performance can also be used to evaluate the model.

[0056] In some optional embodiments, a random masking operation is performed on the text tokenization processing result, and based on the mask position of the masked text, a predicted word and a predicted probability of the masked position being the predicted word are obtained, including: randomly selecting some words in the text tokenization processing result, and replacing some words with the masked text; generating a predicted word probability distribution at the masked position, the predicted word probability distribution includes multiple predicted words and a target predicted word, and the target predicted probability represents the possibility of the target predicted word at the masked position.

[0057] In this embodiment, the replaced words or phrases can be predicted by a masked language model (MLM), so that the similarity calculation model can deeply understand the context information and learn rich language representations, which can be used to calculate the similarity between texts.

[0058] In the MLM task, a certain percentage of words can be randomly selected for masking, for example, 15% of the words can be selected for masking. Specifically, the selected words can be randomly replaced with a mask tag, which can be [MASK]; the original words can also be retained, for example, 10% of the original words remain unchanged; or the selected words can be replaced with another random tag in the dataset.

[0059] Figure 3 FIG. 2 shows a schematic diagram of the random mask operation process in an embodiment of the present invention. Figure 3As shown, in the input sequence 301, [CLS] indicates the beginning of the document, corresponding to the word vector of the first word in the input text, and the sequence number in the sequence is 1, "Let's" is numbered 2, "stick" is numbered 3, "to" is numbered 4, "improvisation" is numbered 5, "in" is numbered 6, "this" is numbered 7, and "skit" is numbered 8. After the random masking operation, 15% of the words are randomly selected for random masking operation, and "improvisation" is replaced with the mask text [MASK]. After being processed by the feedforward neural network (FFNN) and the activation function (Softmax), in the output sequence 302, the probability of predicting the mask word Zyzzyva based on the output of the mask word position is 0%, the probability of improvisation is 10%, and the probability of Aardvark is 0.1%.

[0060] In some optional implementations, the network public opinion text data is subjected to text tokenization processing to obtain text tokenization processing results including: performing word segmentation on Chinese text in the network public opinion text data based on a word segmentation tool, and performing word segmentation on English text in the network public opinion text data based on spaces and / or punctuation marks to obtain word segmentation results; marking the parts of speech of words and / or phrases in the word segmentation results; performing named entity recognition based on the word segmentation results; and obtaining text tokenization processing results, the text tokenization processing results including text, parts of speech and named entity tags corresponding to words, and text, parts of speech and named entity tags corresponding to phrases.

[0061] In this embodiment, the English text in the network public opinion text data can be segmented based on spaces, punctuation marks, or both. Part-of-speech tagging helps understand the structure and semantics of the text. Identifying named entities in the text, such as names of people, places, or institutions, and marking specific tags for the named entities is very important for understanding the key information in the text.

[0062] In some optional embodiments, based on a trained similarity calculation model, a similarity calculation result corresponding to the network public opinion text data to be tested is obtained, including: inputting the first network public opinion text data and the second network public opinion text data into the trained similarity calculation model to obtain a first text vector and a second text vector; calculating the similarity between the first text vector and the second text vector to obtain a similarity calculation result.

[0063] In this embodiment, if the similarity value between the first text vector and the second text vector is close to 1, it indicates that the two vectors are very similar in semantics. If the similarity value between the first text vector and the second text vector is close to 0, it indicates that the two vectors are not similar in semantics.

[0064] In this way, the processing efficiency of online public opinion texts can be improved. Through the automatic learning and extraction of public opinion text features by the similarity calculation model, the efficient calculation of the similarity of online car-hailing public opinion news can be achieved, thereby improving the efficiency and accuracy of public opinion management; based on the similarity calculation model, a large amount of news data can be processed quickly, and accurate evaluation of news similarity can be achieved, meeting the needs of online car-hailing companies in public opinion monitoring; the similarity calculation model has strong generalization ability and robustness, can operate stably in different contexts and conditions, and can reduce the adverse effects of context changes and noise interference.

[0065] In some optional implementations, the aforementioned method for calculating the similarity of online public opinion texts further includes: grouping the online public opinion text data based on the similarity calculation results, and the texts with similarity calculation results greater than a first preset threshold are grouped as the first group.

[0066] In this embodiment, a clustering algorithm can be used to group online public opinion texts, and similar news can be grouped together based on the similarity calculation results to form different news public opinion groups, thereby realizing group display. In this way, it is convenient for users to quickly browse and understand information, identify and classify news more accurately and realistically, provide more convenient, efficient and humane services, and timely discover negative public opinion from repeated and redundant news, respond to marketing strategies as soon as possible, reduce risks, and improve user satisfaction. The output results of the similarity calculation model can also be fed back to the downstream in real time, such as confirming the similarity of the news text or conducting public opinion analysis through system prompts. At the same time, the output results of the similarity calculation model can also be used for public opinion management and operation strategy optimization of online car-hailing companies, and the operation means can be adjusted in time, such as intervening in the automatic allocation of orders or adjusting unit prices.

[0067] In some optional implementations, the aforementioned method for calculating the similarity of network public opinion texts further includes: adjusting the aforementioned similarity calculation model. In the fine-tuning process of the similarity calculation model, the top layers of the BERT model can be adjusted, which can be the last layer or the last few layers, or one or more linear layers or fully connected layers can be added on top of the output of the BERT model to train these newly added layers.

[0068] The method for calculating the similarity of online public opinion texts provided by the embodiment of the present invention can learn more internal text features and news rules, is more sensitive to news texts in different contexts, different expressions and different themes, and has higher recognition accuracy; it has strong generalization ability, scalability and extensibility, and can cover more news scenarios and user needs through one model, which helps online car-hailing companies to effectively manage public opinion, improve operational efficiency, respond to negative public opinion news in a timely manner, and provide users with a better service experience.

[0069] In this embodiment, a device for calculating the similarity of network public opinion text is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0070] Figure 4 The present invention provides a device for calculating the similarity of network public opinion texts. Figure 4 As shown, including:

[0071] The model building module 401 is used to obtain network public opinion text data and train and build a similarity calculation model based on the network public opinion text data.

[0072] The model training module 402 is used to perform text tokenization processing on the network public opinion text data to obtain the text tokenization processing result, perform a random masking operation on the text tokenization processing result, and obtain the predicted word and the predicted probability of the masked position being the predicted word based on the mask position of the masked text; based on the difference between the predicted probability and the text tokenization processing result, construct a loss function, and when the loss function converges, obtain a trained similarity calculation model.

[0073] The similarity calculation module 403 is used to obtain the similarity calculation results corresponding to the network public opinion text data to be tested based on the trained similarity calculation model.

[0074] In some optional implementations, the model training module 402 includes:

[0075] The first model training unit is used to randomly select some words in the text tokenization processing result and replace the some words with mask text.

[0076] The second model training unit is used to generate a predicted word probability distribution at the mask position, where the predicted word probability distribution includes a plurality of predicted words and a target predicted word's target predicted probability, where the target predicted probability represents the possibility of the target predicted word at the mask position.

[0077] In some optional implementations, the model training module 402 further includes:

[0078] The third model training unit is used to segment the Chinese text in the online public opinion text data based on the word segmentation tool, and segment the English text in the online public opinion text data based on spaces and / or punctuation marks to obtain the word segmentation results; mark the parts of speech of words and / or phrases in the word segmentation results; and perform named entity recognition based on the word segmentation results.

[0079] The fourth model training unit is used to obtain a text tokenization processing result, which includes text, part of speech and named entity labels corresponding to words, and text, part of speech and named entity labels corresponding to phrases.

[0080] In some optional implementations, the similarity calculation module 403 includes:

[0081] The first similarity calculation unit is used to input the first network public opinion text data and the second network public opinion text data into the trained similarity calculation model to obtain the first text vector and the second text vector.

[0082] The second similarity calculation unit is used to calculate the similarity between the first text vector and the second text vector to obtain a similarity calculation result.

[0083] In some optional embodiments, the aforementioned device for calculating the similarity of online public opinion texts also includes: a grouping module, which is used to group the online public opinion text data based on the similarity calculation results, and the similarity calculation results greater than the first preset threshold are the first group.

[0084] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0085] The device for calculating the similarity of online public opinion texts in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0086] The embodiment of the present invention also provides a computer device having the above Figure 4The device shown is used for calculating the similarity of network public opinion texts.

[0087] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0088] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0089] The aforementioned memory 20 stores instructions executable by at least one processor 10, so that the aforementioned at least one processor 10 executes the method shown in the above embodiment.

[0090] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0091] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0092] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0093] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as a light emitting diode) and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0094] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0095] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0096] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for calculating the similarity of network public opinion texts, characterized in that: The method comprises: Acquire network public opinion text data, and train and construct a similarity calculation model based on the network public opinion text data; The network public opinion text data is subjected to text tokenization processing to obtain a text tokenization processing result, a random masking operation is performed on the text tokenization processing result, and based on the mask position of the masked text, a predicted word and a predicted probability that the mask position is the predicted word are obtained; based on the difference between the predicted probability and the text tokenization processing result, a loss function is constructed, and when the loss function converges, a trained similarity calculation model is obtained; Based on the trained similarity calculation model, a similarity calculation result corresponding to the network public opinion text data to be tested is obtained.

2. The method according to claim 1, characterized in that Based on the difference between the predicted probability and the text tokenization processing result, a loss function is constructed, including: The loss function is constructed by calculating the cross entropy loss between the predicted probability and the text tokenization result. The cross entropy loss H(y, y) ′ ) is calculated by the following formula: Among them, y i is the one-hot encoding of the i-th component of the real data in the text tokenization processing result. If the real data is of the i-th category, then y i =1, otherwise, y i =0;y i ′ is the predicted probability of the i-th category.

3. The method according to claim 1 or 2, characterized in that: Performing a random masking operation on the text tokenization processing result, and obtaining a predicted word and a predicted probability that the masked position is the predicted word based on a masked position of the masked text, including: Randomly selecting some words in the text tokenization processing result, and replacing the some words with the mask text; A predicted word probability distribution is generated at the mask position, wherein the predicted word probability distribution includes a plurality of predicted words and a target predicted word, and the target predicted probability represents the possibility of the target predicted word at the mask position.

4. The method according to claim 1, characterized in that: The network public opinion text data is subjected to text tokenization processing, and the text tokenization processing results obtained include: For the Chinese text in the network public opinion text data, the Chinese text is segmented based on a word segmentation tool, and for the English text in the network public opinion text data, the English text is segmented based on spaces and / or punctuation marks to obtain a word segmentation result; Marking the parts of speech of the words and / or phrases in the word segmentation results; Perform named entity recognition based on the word segmentation result; The text tokenization processing result is obtained, and the text tokenization processing result includes the text, part of speech and named entity tag corresponding to the word, and the text, part of speech and named entity tag corresponding to the phrase.

5. The method according to claim 1, characterized in that Based on the trained similarity calculation model, a similarity calculation result corresponding to the network public opinion text data to be tested is obtained, including: Inputting the first network public opinion text data and the second network public opinion text data into the trained similarity calculation model to obtain a first text vector and a second text vector; The similarity between the first text vector and the second text vector is calculated to obtain the similarity calculation result.

6. The method according to claim 1, characterized in that The method further comprises: The network public opinion text data is grouped based on the similarity calculation result, and the data having a similarity calculation result greater than a first preset threshold is grouped as a first group.

7. A device for calculating the similarity of network public opinion texts, characterized in that: The device comprises: A model building module is used to obtain network public opinion text data and train and build a similarity calculation model based on the network public opinion text data; A model training module is used to perform text tokenization processing on the network public opinion text data to obtain a text tokenization processing result, perform a random masking operation on the text tokenization processing result, and obtain a predicted word and a predicted probability that the mask position is the predicted word based on a mask position of the masked text; based on the difference between the predicted probability and the text tokenization processing result, a loss function is constructed, and when the loss function converges, a trained similarity calculation model is obtained; The similarity calculation module is used to obtain the similarity calculation results corresponding to the network public opinion text data to be tested based on the trained similarity calculation model.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for calculating the similarity of network public opinion texts as described in any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for calculating the similarity of network public opinion texts according to any one of claims 1 to 6.

10. A computer program product, characterized in that It includes computer instructions, which are used to enable a computer to execute the method for calculating the similarity of network public opinion texts as described in any one of claims 1 to 6.

Images