Text similarity calculation method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides a text similarity calculation method, device, electronic device and storage medium, which belongs to the field of artificial intelligence technology. The method includes: obtaining the original text to be calculated; using a pre-trained text segmentation model to perform segmentation processing on the original text to obtain multiple text segments; using a pre-trained target vocabulary model to perform position identification on each text segment to obtain the target position of each text segment; encoding each text segment according to the target position to obtain a text segment vector; inputting the text segment vector into a pre-trained comparison model to perform matrix multiplication of the text segment vector with the reference word embedding matrix in the comparison model to obtain a target word embedding vector; performing similarity calculation on multiple target word embedding vectors to obtain the similarity value between each two text segments. The embodiment of the present application can more accurately calculate the correlation between text segments.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text similarity calculation method, device, electronic device, and storage medium. Background Art

[0002] With the advancement of computer technology, many businesses require the use of natural language processing (NLP), such as search engines and intelligent services. When performing NLP, text similarity calculations are often performed. However, current calculation methods often suffer from low accuracy. Therefore, developing a method to improve the accuracy of text similarity calculations has become a pressing technical challenge. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to provide a text similarity calculation method, device, electronic device and storage medium, aiming to improve the accuracy of text similarity calculation.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for calculating text similarity, the method comprising:

[0005] Get the original text to be calculated;

[0006] Using a pre-trained text segmentation model to perform word segmentation processing on the original text to obtain multiple text segments;

[0007] Using a pre-trained target vocabulary model to perform position recognition on each of the text word segments to obtain a target position of each of the text word segments;

[0008] Encoding each of the text segments according to the target position to obtain a text segment vector;

[0009] Inputting the text segment vector into a pre-trained contrast model to perform matrix multiplication of the text segment vector with a reference word embedding matrix in the contrast model to obtain a target word embedding vector;

[0010] Similarity calculation is performed on the multiple target word embedding vectors to obtain a similarity value between every two text word segments.

[0011] In some embodiments, the step of using a pre-trained target vocabulary model to identify the position of each text segment to obtain the target position of each text segment includes:

[0012] Extracting elements from each text segment using a preset function in the target vocabulary model to obtain an element value for each text segment;

[0013] The position of the text segment is identified according to the element value to obtain the target position of the text segment.

[0014] In some embodiments, before the step of using a pre-trained target vocabulary model to identify the position of each text word segment to obtain the target position of each text word segment, the method further includes pre-training the target vocabulary model, specifically including:

[0015] Obtain reference text data;

[0016] Performing word segmentation processing on the reference text data using an initial word segmentation model to obtain reference word segment data;

[0017] Dividing the reference segment data into a training set, a test set, and a validation set according to a preset ratio;

[0018] Using the training set to train the initial model to obtain a current vocabulary model;

[0019] The current vocabulary model is verified using the test set and the validation set to obtain the target vocabulary model.

[0020] In some embodiments, the step of encoding each text segment according to the target position to obtain a text segment vector includes:

[0021] According to the target position, each text segment is normalized to obtain a standard segment;

[0022] The standard word segment is one-hot encoded to obtain a text word segment vector.

[0023] In some embodiments, the step of inputting the text segment vector into a pre-trained contrastive model to perform matrix multiplication on the text segment vector and a reference word embedding matrix in the contrastive model to obtain a target word embedding vector includes:

[0024] Inputting the text segment vector into the comparison model to perform matrix multiplication of the text segment vector with the reference word embedding matrix to obtain a plurality of basic word embedding vectors;

[0025] Mapping is performed on the basic word embedding vector to obtain a target word embedding vector.

[0026] In some embodiments, before the step of inputting the text segment vector into a pre-trained contrastive model to perform matrix multiplication on the text segment vector and a reference word embedding matrix in the contrastive model to obtain a target word embedding vector, the method further includes training the contrastive model, specifically comprising:

[0027] Get sample data;

[0028] Performing data enhancement processing on the sample data to obtain positive example pairs;

[0029] Inputting the positive example pair into the comparison model;

[0030] Calculating a first similarity of the positive example pair and a second similarity of the negative example pair using a loss function of the comparison model;

[0031] The loss function of the comparison model is optimized according to the first similarity and the second similarity to update the comparison model.

[0032] In some embodiments, the step of performing similarity calculation on the plurality of target word embedding vectors to obtain a similarity value between each two text segments includes:

[0033] The cosine similarity algorithm is used to calculate the similarity of multiple target word embedding vectors to obtain the similarity value between each two text segments.

[0034] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a text similarity calculation device, the device comprising:

[0035] The text acquisition module is used to obtain the original text to be calculated;

[0036] A word segmentation module is used to perform word segmentation processing on the original text using a pre-trained text word segmentation model to obtain multiple text segments;

[0037] A position identification module, configured to identify the position of each text word segment using a pre-trained target word library model to obtain a target position of each text word segment;

[0038] an encoding module, configured to encode each of the text segments according to the target position to obtain a text segment vector;

[0039] A comparison module, configured to input the text segment vector into a pre-trained comparison model so as to perform matrix multiplication of the text segment vector with a reference word embedding matrix in the comparison model to obtain a target word embedding vector;

[0040] The similarity calculation module is used to perform similarity calculation on the multiple target word embedding vectors to obtain the similarity value between each two text segments.

[0041] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored on the memory and runnable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the method described in the first aspect above is implemented.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium for computer-readable storage, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.

[0043] The text similarity calculation method, device, electronic device and storage medium proposed in the present application obtain the original text to be calculated, and use a pre-trained text segmentation model to perform word segmentation processing on the original text to obtain multiple text segments. This method can achieve fragmentation processing of the original text, making it easier to extract the required text segments; then use a pre-trained target vocabulary model to identify the position of each text segment, obtain the target position of each text segment, and more accurately identify the position of each text segment. In this way, each text segment can be further encoded according to the target position to obtain a text segment vector, and the text segment vector is input into a pre-trained comparison model. The text segment vector is matrix-multiplied with the reference word embedding matrix in the comparison model to obtain a target word embedding vector. Finally, similarity calculation is performed on multiple target word embedding vectors to obtain the similarity value between each two text segments. The comparison model can effectively solve the problem of uniform distribution of segment vectors, thereby improving the accuracy of similarity calculation. This method can more accurately determine the correlation between text segments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of a method for calculating text similarity provided by an embodiment of the present application;

[0045] Figure 2 yes Figure 1 Flowchart of step S103 in FIG.

[0046] Figure 3 is another flow chart of the text similarity calculation method provided in an embodiment of the present application;

[0047] Figure 4 yes Figure 1 Flowchart of step S104 in FIG.

[0048] Figure 5 yes Figure 1Flowchart of step S105 in FIG.

[0049] Figure 6 is another flow chart of the text similarity calculation method provided in an embodiment of the present application;

[0050] Figure 7 Schematic diagram of the structure of the text similarity calculation device provided in an embodiment of the present application;

[0051] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0055] First, let’s analyze some of the terms used in this application:

[0056] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0057] Natural language processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics, often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and discourse understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0058] Information Extraction (NER): A text processing technology that extracts specified types of entity, relationship, event, and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data is all text information extraction. Of course, the information extracted by text information extraction technology can be of various types.

[0059] Entity: A distinct, independently existing thing. Examples include a person, a city, a plant, or a commodity. Everything in the world is composed of specific things, referred to as entities. Entities are the most basic elements in a knowledge graph, and different entities have different relationships with each other.

[0060] Concept: A collection of entities of a certain type.

[0061] Semantic class (concept): A collection of entities with similar characteristics, such as countries, ethnic groups, books, computers, etc. Concepts mainly refer to collections, categories, object types, and types of things, such as people and geography.

[0062] Self-supervised learning: Self-supervised learning primarily uses auxiliary tasks (pretext) to mine self-supervised information from large-scale unsupervised data. This constructed supervised information is then used to train the network, thereby learning representations that are valuable for downstream tasks. In other words, the supervised information in self-supervised learning is not manually annotated, but rather automatically constructed by the algorithm from large-scale unsupervised data for supervised learning or training.

[0063] Contrastive learning is a type of self-supervised learning that does not rely on manually annotated category labels but instead uses the data itself as supervision. Contrastive learning is a method for deep learning models to characterize similar and different objects. Using contrastive learning, machine learning models can be trained to distinguish between similar and different images. Self-supervised learning in the image domain is divided into two types: generative self-supervised learning and discriminative self-supervised learning. Contrastive learning employs a typical discriminative self-supervised learning approach. The core principle of contrastive learning is to automatically construct similar and dissimilar instances, namely positive and negative samples, and learn to compare these positive and negative samples in feature space. This results in similar instances being closer together and dissimilar instances being farther apart, resulting in greater differences. The model representations obtained through this learning process can then be used to perform downstream tasks and fine-tuned on smaller labeled datasets, thus achieving unsupervised model learning. The guiding principle of contrastive learning is to automatically construct similar and dissimilar instances, obtain a learning model through learning, and use this model to make similar instances closer in the projected space, while dissimilar instances are farther apart in the projected space.

[0064] Embedding: An embedding is a vector representation that represents an object using a low-dimensional vector. This object can be a word, a product, a movie, or anything else. The nature of this embedding vector is that it allows objects corresponding to closely spaced vectors to have similar meanings. For example, the distance between embedding(Avengers) and embedding(Iron Man) will be very close, but the distance between embedding(Avengers) and embedding(Gone with the Wind) will be farther. Embedding is essentially a mapping from semantic space to vector space, while simultaneously preserving the relationship between the original samples in the semantic space as much as possible. For example, two semantically similar words will be positioned relatively close in the vector space. Embedding can encode objects using low-dimensional vectors while preserving their meaning. It is often used in machine learning. When building machine learning models, objects are encoded as low-dimensional dense vectors and then passed to the DNN to improve efficiency.

[0065] Batch: Batch size is a hyperparameter that defines the number of examples to process before updating internal model parameters. This means it controls the number of training examples before updating the model's internal parameters. A training dataset can be divided into one or more batches. When all training examples are used to create a single batch, the learning algorithm is called batch gradient descent; when the batch size is the size of one example, the learning algorithm is called stochastic gradient descent; and when the batch size exceeds one example and is smaller than the size of the training dataset, the learning algorithm is called mini-batch gradient descent. The batch size is the number of examples processed before updating the model.

[0066] Data augmentation: Data augmentation is primarily used to prevent overfitting and optimize datasets when they are small. Data augmentation can increase the amount of training data, improve the model's generalization capabilities, and increase noise data, thereby enhancing the model's robustness. Data augmentation can be divided into two categories: offline augmentation and online augmentation. Offline augmentation directly processes the dataset, converting the number of data points to an augmentation factor x the number of original datasets. Offline augmentation is often used when datasets are very small. Online augmentation is primarily used to enhance batches of data after they are acquired, such as by performing rotations, translations, and folding. Because some datasets cannot tolerate linear growth, online augmentation is often used for larger datasets. Many machine learning frameworks already support online augmentation and can use GPUs for optimized computation.

[0067] Dropout: Dropout is a technique to prevent model overfitting. It means that during the training process of a deep learning network, neural network units are temporarily discarded from the network with a certain probability, so that the model can be more robust because it will not rely too much on certain local features (because local features may be discarded).

[0068] Masking is a common operation in deep learning. Simply put, masking is equivalent to adding a mask to the original tensor, thereby blocking or selecting specific elements. Therefore, it is often used to construct tensor filters. The linear activation function ReLU (which performs a simple binary split based on the positive and negative range of the output) and the dropout mechanism (which performs a binary split based on probability) can both be understood as generalized masking operations.

[0069] Encoder: Encoding is to convert the input sequence into a vector of fixed length; decoding is to convert the previously generated fixed vector into an output sequence; the input sequence can be text, voice, image, video; the output sequence can be text or image.

[0070] Back propagation: The general principle of back propagation is: input the training set data into the input layer of the neural network, pass through the hidden layer of the neural network, and finally reach the output layer of the neural network and output the result; since there is an error between the output result of the neural network and the actual result, the error between the estimated value and the actual value is calculated, and the error is backpropagated from the output layer to the hidden layer until it propagates to the input layer; in the process of back propagation, the values ​​of various parameters are adjusted according to the error; the above process is continuously iterated until convergence.

[0071] With the advancement of computer technology, many businesses require the use of natural language processing (NLP), such as search engines and intelligent services. When performing NLP, text similarity calculations are often performed. However, current calculation methods often suffer from low accuracy. Therefore, developing a method for calculating text similarity and improving its accuracy has become a pressing technical challenge.

[0072] Based on this, embodiments of the present application provide a text similarity calculation method, device, electronic device, and storage medium, which can improve the accuracy of similarity calculation.

[0073] The text similarity calculation method, device, electronic device, and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the text similarity calculation method in the embodiments of the present application is described.

[0074] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0075] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] The text similarity calculation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The text similarity calculation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the text similarity calculation method, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0078] Figure 1 This is an optional flowchart of the text similarity calculation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0079] Step S101, obtaining the original text to be calculated;

[0080] Step S102: using a pre-trained text segmentation model to perform word segmentation processing on the original text to obtain multiple text segments;

[0081] Step S103, using a pre-trained target vocabulary model to perform position recognition on each text segment to obtain a target position of each text segment;

[0082] Step S104: encoding each text segment according to the target position to obtain a text segment vector;

[0083] Step S105: input the text segment vector into a pre-trained contrast model to perform matrix multiplication of the text segment vector and the reference word embedding matrix in the contrast model to obtain a target word embedding vector;

[0084] Step S106: perform similarity calculation on multiple target word embedding vectors to obtain a similarity value between every two text word segments.

[0085] In step S101 of some embodiments, a web crawler can be written to crawl data in a targeted manner after setting up a data source to obtain the original text to be calculated. It should be noted that the original text is a natural language text.

[0086] In step S102 of some embodiments, the pre-trained text segmentation model may include Jieba segmenter; or other segmentation software, such as hanlp, etc. Taking Jieba segmenter as an example of the text segmentation model, step S102 specifically includes:

[0087] The pre-trained Jieba word segmenter is used to segment the original text to obtain text segments.

[0088] Specifically, when using the Jieba word segmenter for word segmentation processing, the directed acyclic graph corresponding to the original text is first generated by comparing the dictionary in the Jieba word segmenter, and then the shortest path on the directed acyclic graph is found according to the preset selection mode and dictionary. The original text is intercepted according to the shortest path, or the original text is directly intercepted to obtain text segments.

[0089] Furthermore, for text segments not in the dictionary, HMMs (Hidden Markov Models) can be used for new word discovery. Specifically, the character positions B, M, E, and S in the text segment are used as hidden states, and the characters are observed states. B / M / E / S represent occurrences at the beginning, middle, and end of a word, respectively, and as single characters in a word. A dictionary file is used to store the representation probability matrix, initial probability vector, and transition probability matrix between characters. The Viterbi algorithm is then used to solve for the maximum possible hidden state, thereby obtaining the text field.

[0090] In step S102 of some other embodiments, it is also necessary to perform part-of-speech tagging on the text segment, that is, to tag the text segment with part-of-speech according to preset part-of-speech categories to obtain a text segment containing a part-of-speech category label, wherein the preset part-of-speech categories include names, verbs, modifiers, adjectives, etc.

[0091] Through step S102 of the above embodiment, word segmentation processing of the original text can be achieved, making it easier to extract the required text segments.

[0092] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S202:

[0093] Step S201, extracting elements from each text segment using a preset function in the target vocabulary model to obtain element values ​​for each text segment;

[0094] Step S202 : performing position recognition on the text segment according to the element value to obtain the target position of the text segment.

[0095] Specifically, the target vocabulary model can be an encoding model, including at least one encoder. Furthermore, to extract element values ​​from text segments, the target vocabulary model is also pre-configured with an index function. Since the index function can return element values ​​in a table or array, in step S201, the element values ​​of each text segment are extracted using the array-based index function to obtain the element values ​​of the text segment. The element values ​​of a text segment include index values ​​for the row and column numbers of the text segment. Therefore, by searching the row and column numbers of the text segment using the index function, the text segment at a specified position can be retrieved. In step S202, the row and column numbers of the text segment are searched using the index function, traversing each text field in the original text to generate a position sequence table for the text segments. This position sequence table can reflect the correspondence between the text fields and the row and column numbers (element values). This allows the target position of the text segment to be determined based on the element value, thereby enabling relatively accurate position identification of each text segment.

[0096] It should be noted that other functions may be preset in the target vocabulary model to extract element values ​​of text segments, not limited to the index function.

[0097] See also Figure 3 In some embodiments, before step S103, the method further includes pre-training the target vocabulary model, which may specifically include but is not limited to steps S301 to S305:

[0098] Step S301, obtaining reference text data;

[0099] Step S302: Using the initial word segmentation model to perform word segmentation processing on the reference text data to obtain reference word segment data;

[0100] Step S303, dividing the reference segment data into a training set, a test set, and a validation set according to a preset ratio;

[0101] Step S304: training the initial model using the training set to obtain the current vocabulary model;

[0102] Step S305: Verify the current vocabulary model using the test set and the validation set to obtain the target vocabulary model.

[0103] Specifically, in step S301, reference text data can be obtained through a web crawler or other means. In step S302, the initial word segmentation model can be a Jieba word segmenter, and the Jieba word segmenter is used to segment the reference text data to obtain reference word segment data. The specific process of segmenting the reference text data is basically the same as the process of segmenting the original text in step S102 above, and will not be repeated here.

[0104] Then, step S303 can be executed to divide the reference segment data into a training set, a validation set, and a test set according to a preset ratio. It should be noted that the preset ratio can be set according to actual needs. For example, the reference segment data can be divided into a training set, a validation set, and a test set in a ratio of 7:2:1, and the reference segment data of these three data sets are annotated to obtain labeled reference segment data.

[0105] Specifically, after executing step S303, step S304 may be executed to input the reference segment data of the training set into the initial model, perform model training on the initial model, and obtain the current vocabulary model, wherein the initial model includes an encoder.

[0106] Finally, step S305 is executed to input the reference word segment data of the validation set into the current vocabulary model, observe the convergence of the validation set accuracy, monitor whether the current vocabulary model is overfitting, and adjust the model parameters of the current vocabulary module. Furthermore, step S305 includes: verifying the current vocabulary model using the test set and the validation set to obtain the optimal vocabulary model, optimizing the optimal vocabulary model to obtain the target vocabulary model. In step S305, by comparing the comprehensive index MAP of the vocabulary models at different times, the current vocabulary model with the highest MAP index is selected as the optimal vocabulary model, and then the optimal vocabulary model is optimized to obtain the target vocabulary model. The target vocabulary model is used to identify the position of each text segment to obtain the target position of each text segment, which can improve recognition accuracy.

[0107] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S402:

[0108] Step S401, normalizing each text segment according to the target position to obtain a standard segment;

[0109] Step S402: One-hot encode the standard word segment to obtain a text word segment vector.

[0110] Specifically, the target position is the index position of the text segment. In step S401 of some embodiments, each text segment is extracted from the original text according to the index position of each text segment, and each text segment is linearly scaled to [-1, 1], or each text segment is scaled to a mean of 0 and a variance of 1, so as to achieve normalization processing for each text segment and obtain a standard segment.

[0111] It should be noted that one-hot encoding, also known as single-bit encoding, uses an N-bit state register to encode N states. Each state has its own independent register bit, and only one state is valid at any time.

[0112] In step S402 of some embodiments, the length of the standard word segment can be represented as a vector through one-hot encoding to obtain multiple text word segment vectors. For example, assuming that an original text consists of 3 text word segments, the index positions of these 3 text word segments can be obtained through the above steps. One-hot encoding is to use a vector of length V to represent each text word segment, and this V is the number of dictionary words corresponding to the text word segment in the target vocabulary model. The vector marks the index position of the text word segment in the original text as 1, and the others as 0. Assuming that a sentence consists of 3 text word segments, there will be 3 1s in this vector, and the position of this 1 can correspond to the index position of the text word segment.

[0113] Through step S104 of the above embodiment, each text segment can be encoded according to the target position to obtain a text segment vector, and the target word embedding vector can be obtained through the text segment vector.

[0114] See also Figure 5 In some embodiments, step S105 may also include but is not limited to steps S501 to S502:

[0115] Step S501: inputting the text segment vector into the comparison model to perform matrix multiplication of the text segment vector and the reference word embedding matrix to obtain multiple basic word embedding vectors;

[0116] Step S502: Map the basic word embedding vector to obtain the target word embedding vector.

[0117] Specifically, step S501 is performed to completely fix the values ​​of the reference word embedding matrix within the contrastive model by training the contrastive model, and other model parameters of the contrastive model are also fixed. Therefore, when a text segment vector is input into the contrastive model, the fixed reference word embedding matrix can be used to perform matrix multiplication with each text segment vector to obtain a basic word embedding vector.

[0118] Then, step S502 can be performed to map the basic word embedding vector using the fixed MLP network in the comparison model to obtain the target word embedding vector. The MLP network includes a linear layer, a ReLu activation function, and a linear layer.

[0119] See also Figure 6 In some embodiments, before step S105, the method further includes training a comparison model, which may specifically include but is not limited to steps S601 to S605:

[0120] Step S601, obtaining sample data;

[0121] Step S602: perform data enhancement processing on the sample data to obtain positive example pairs;

[0122] Step S603, inputting the positive example pair into the comparison model;

[0123] Step S604, calculating the first similarity of the positive example pair and the second similarity of the negative example pair by comparing the loss function of the model;

[0124] Step S605 : Optimize the loss function of the comparison model according to the first similarity and the second similarity to update the comparison model.

[0125] Specifically, the sample data is first mapped to the embedding space and the sample data is represented by a vector, so that the initial embedding data (i.e., initial embedding data) can be obtained. The initial embedding data includes positive sample data and negative sample data.

[0126] In step S602 of some embodiments, the initial embedded data is subjected to data enhancement processing using a dropout mask mechanism. In the embodiments of the present application, the dropout mask mechanism replaces the traditional data enhancement method, i.e., the two vectors obtained by inputting the same sample data twice into the dropout encoder are used as positive example pairs for comparative learning. This is effective enough because, for example, each dropout within BERT randomly generates a different dropout mask. Therefore, it is only necessary to input the same sample data (i.e., the initial embedded data of this embodiment) into the simCSE model twice, and the two vectors obtained are the result of applying two different dropout masks. It can be understood that the dropout mask is a random network model, a mask for the model parameter W, which plays a role in preventing overfitting.

[0127] In a batch, the data obtained after data augmentation (i.e., the first vector and the second vector) is a positive pair, and the other data without data augmentation is a negative pair. In this embodiment of the present application, a portion of the initial embedded data in a batch can be processed through data augmentation to obtain positive pairs, and the remaining portion of the initial embedded data can be used as negative pairs.

[0128] In some embodiments, positive pairs are generated by randomly sampling a dropout mask.

[0129] In some specific application scenarios, during the contrastive learning phase, a typical intra-batch contrastive learning method is used to perform data augmentation within the batch. This means that the complete initial embedding data obtained above is augmented to make the two positive samples (the first sample data and the second sample data) different. This embodiment of the present application directly uses dropout as data augmentation, i.e., it generates positive pairs by randomly sampling the dropout mask. More specifically, the same sample is repeatedly input twice into the same dropout encoder (i.e., the same first sample data and the second sample data are respectively input into the dropout encoder for data augmentation), thereby obtaining two different representation vectors. (first vector) and (Second vector), take the first vector and the second vector as a positive example pair< , >. In practice, < , >The sentence vectors are different, but because the input sentences are the same, the semantic expectations of the final sentence vectors are the same, so they are used as positive examples to allow the model to narrow the distance between them.

[0130] In step S604 of some embodiments, the first similarity and the second similarity are both cosine similarities, and the loss function of the comparison model is optimized according to the first similarity and the second similarity, which may include but is not limited to:

[0131] The first similarity is maximized to a first value and the second similarity is minimized to a first value to optimize the loss function; wherein the first similarity is the numerator of the loss function, the first similarity and the second similarity are the denominator of the loss function, the first value is 1, and the second value is 0. In the loss function, the numerator is the first similarity corresponding to the positive example pair, the denominator is the first similarity and the second similarity of all negative example pairs, and then the molecular formula value composed of the numerator and denominator is wrapped in -log(), so that the numerator is maximized and the denominator is minimized to minimize the loss function. In this embodiment, minimizing the loss function infoNCE loss is to maximize the numerator and minimize the denominator, that is, to maximize the first similarity of the positive example pair and minimize the second similarity of the negative example pair, and to minimize the loss function to optimize the loss function. More specifically, the loss function is shown in formula (1):

[0132] Formula (1)

[0133] in, yes The transpose of It is the original sample, is a positive example, It is a single negative sample, and then all negative samples are accumulated. The denominator includes one positive sample and N-1 negative samples;

[0134] This loss function represents the loss of sample N. In this loss function, the numerator is the similarity of the positive pair, and the denominator is the similarity of the positive pair and all negative pairs. This value is then wrapped in -log(). By maximizing the numerator and minimizing the denominator, the loss function can be minimized.

[0135] It should be noted that the similarity between the positive example pair (first similarity) and the negative example pair (second similarity) meets the following conditions:

[0136]

[0137] From the above formula, we can see that this method needs to satisfy the following conditions: the similarity of the positive pair is greater than or equal to the similarity of the negative pair, where x+ refers to data similar to x, that is, the positive sample pair data; here x- refers to data dissimilar to x, that is, the negative sample pair data. is a positive example, .

[0138] Furthermore, the preset metric function is:

[0139] Formula (3)

[0140] Formula (4)

[0141] Score is a metric function used to evaluate the similarity between two features. The default metric function uses the dot product as the score function.

[0142] Specifically, in step S605 of some embodiments, optimizing the loss function of the comparison model according to the first similarity and the second similarity may include but is not limited to:

[0143] Backpropagation is performed according to the loss function to update the loss parameters of the loss function to optimize the loss function.

[0144] In the embodiment of the present application, backpropagation is performed based on the loss function to update the comparison model by optimizing the loss function, thereby updating the internal parameters of the comparison model (i.e., the loss parameters). It is understood that the backpropagation principle can apply conventional backpropagation principles, and the embodiment of the present application is not limited thereto.

[0145] In some embodiments, step S106 may include but is not limited to the following steps:

[0146] The cosine similarity algorithm is used to calculate the similarity of multiple target word embedding vectors to obtain the similarity value between each two text segments.

[0147] Specifically, when calculating the similarity value between each two text segments, assuming that the target embedding vector of one text segment is u and the target embedding vector of the other text segment is v, the similarity value of the two text segments is calculated according to the formula of the cosine similarity algorithm (as shown in Formula 5).

[0148] Formula (5)

[0149] The embodiment of the present application obtains the original text to be calculated, and uses a pre-trained text segmentation model to perform segmentation processing on the original text to obtain multiple text segments. This method can achieve fragmentation processing of the original text, making it easier to extract the required text segments; then, the pre-trained target vocabulary model is used to identify the position of each text segment, and the target position of each text segment is obtained, which can more accurately identify the position of each text segment. In this way, each text segment can be further encoded according to the target position to obtain a text segment vector, and the text segment vector is input into the pre-trained comparison model, so that the text segment vector is matrix-multiplied with the reference word embedding matrix in the comparison model to obtain a target word embedding vector. Finally, the text similarity calculation is performed on the multiple target word embedding vectors to obtain the similarity value between each two text segments. The comparison model can effectively solve the problem of uniform distribution of segment vectors, thereby improving the accuracy of similarity calculation. This method can more accurately determine the correlation between text segments.

[0150] See also Figure 7 The present application also provides a text similarity calculation device that can implement the above-mentioned text similarity calculation method. The device includes:

[0151] The text acquisition module 701 is used to acquire the original text to be calculated;

[0152] The word segmentation module 702 is used to perform word segmentation processing on the original text using a pre-trained text word segmentation model to obtain multiple text segments;

[0153] A position identification module 703 is used to identify the position of each text word segment using a pre-trained target word library model to obtain a target position of each text word segment;

[0154] An encoding module 704 is used to encode each text segment according to the target position to obtain a text segment vector;

[0155] A comparison module 705 is configured to input the text segment vector into a pre-trained comparison model to perform matrix multiplication of the text segment vector with a reference word embedding matrix in the comparison model to obtain a target word embedding vector;

[0156] The similarity calculation module 706 is used to perform similarity calculation on multiple target word embedding vectors to obtain a similarity value between each two text segments.

[0157] The specific implementation of the text similarity calculation device is substantially the same as the specific embodiment of the above-mentioned text similarity calculation method, and will not be described in detail here.

[0158] The present application also provides an electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, the above-described text similarity calculation method is implemented. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0159] See also Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0160] The processor 801 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0161] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called by the processor 801 to execute the text similarity calculation method of the embodiments of this application.

[0162] Input / output interface 803, used to implement information input and output;

[0163] Communication interface 804, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); and

[0164] Bus 805 , which transmits information between various components of the device (e.g., processor 801 , memory 802 , input / output interface 803 , and communication interface 804 );

[0165] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0166] An embodiment of the present application also provides a computer-readable storage medium for computer-readable storage. The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned text similarity calculation method.

[0167] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0169] It will be understood by those skilled in the art that Figure 1-6 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0171] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0172] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0173] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0178] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for calculating text similarity, characterized in that: The method comprises: Get the original text to be calculated; Using a pre-trained text segmentation model to perform word segmentation processing on the original text to obtain multiple text segments; Using a pre-trained target vocabulary model to perform position recognition on each of the text word segments to obtain a target position of each of the text word segments; Encoding each of the text segments according to the target position to obtain a text segment vector; Inputting the text segment vector into a pre-trained contrast model to perform matrix multiplication of the text segment vector with a reference word embedding matrix in the contrast model to obtain a target word embedding vector; Performing similarity calculation on the target word embedding vectors to obtain a similarity value between each two text segments; The method of using a pre-trained target vocabulary model to identify the position of each text segment to obtain the target position of each text segment includes: Extracting elements from each text segment using an array-based index function in the target vocabulary model to obtain an element value for each text segment, wherein the element value for each text segment includes index values ​​for the row and column numbers of the text segment; The row number and column number of the text segment are searched through the index function, each text field in the original text is traversed, and a position sequence table of the text segment is generated, so that the position of the text segment can be identified according to the element value to obtain the target position of the text segment. The position sequence table is used to reflect the correspondence between the text field and the row number and element value, and the target position is the index position of the text segment.

2. The text similarity calculation method according to claim 1, characterized in that Before the step of using a pre-trained target vocabulary model to identify the position of each text word segment to obtain the target position of each text word segment, the method further includes pre-training the target vocabulary model, specifically including: Obtain reference text data; Performing word segmentation processing on the reference text data using an initial word segmentation model to obtain reference word segment data; Dividing the reference segment data into a training set, a test set, and a validation set according to a preset ratio; Using the training set to train the initial model to obtain a current vocabulary model; The current vocabulary model is verified using the test set and the validation set to obtain the target vocabulary model.

3. The text similarity calculation method according to claim 1, characterized in that The step of encoding each text segment according to the target position to obtain a text segment vector includes: According to the target position, each text segment is normalized to obtain a standard segment; The standard word segment is one-hot encoded to obtain a text word segment vector.

4. The text similarity calculation method according to claim 1, characterized in that: The step of inputting the text segment vector into a pre-trained contrast model to perform matrix multiplication on the text segment vector and a reference word embedding matrix in the contrast model to obtain a target word embedding vector includes: Inputting the text segment vector into the comparison model to perform matrix multiplication of the text segment vector with the reference word embedding matrix to obtain a plurality of basic word embedding vectors; Mapping is performed on the basic word embedding vector to obtain a target word embedding vector.

5. The text similarity calculation method according to claim 1, characterized in that: Before the step of inputting the text segment vector into a pre-trained contrastive model to perform matrix multiplication on the text segment vector and a reference word embedding matrix in the contrastive model to obtain a target word embedding vector, the method further includes training the contrastive model, specifically comprising: Get sample data; Performing data enhancement processing on the sample data to obtain positive example pairs; Inputting the positive example pair into the comparison model; Calculating a first similarity of the positive example pair and a second similarity of the negative example pair using a loss function of the comparison model; The loss function of the comparison model is optimized according to the first similarity and the second similarity to update the comparison model.

6. The text similarity calculation method according to any one of claims 1 to 5, characterized in that: The step of performing similarity calculation on the plurality of target word embedding vectors to obtain a similarity value between each two text segments includes: The cosine similarity algorithm is used to calculate the similarity of multiple target word embedding vectors to obtain the similarity value between each two text segments.

7. A text similarity calculation device, characterized in that: The device comprises: The text acquisition module is used to obtain the original text to be calculated; A word segmentation module is used to perform word segmentation processing on the original text using a pre-trained text word segmentation model to obtain multiple text segments; A position identification module, configured to identify the position of each text word segment using a pre-trained target word library model to obtain a target position of each text word segment; an encoding module, configured to encode each of the text segments according to the target position to obtain a text segment vector; A comparison module, configured to input the text segment vector into a pre-trained comparison model so as to perform matrix multiplication of the text segment vector with a reference word embedding matrix in the comparison model to obtain a target word embedding vector; A similarity calculation module is used to perform similarity calculation on the multiple target word embedding vectors to obtain a similarity value between each two text segments; The method of using a pre-trained target vocabulary model to identify the position of each text segment to obtain the target position of each text segment includes: Extracting elements from each text segment using an array-based index function in the target vocabulary model to obtain an element value for each text segment, wherein the element value for each text segment includes index values ​​for the row and column numbers of the text segment; The row number and column number of the text segment are searched through the index function, each text field in the original text is traversed, and a position sequence table of the text segment is generated, so that the position of the text segment can be identified according to the element value to obtain the target position of the text segment. The position sequence table is used to reflect the correspondence between the text field and the row number and element value, and the target position is the index position of the text segment.

8. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the text similarity calculation method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium for computer-readable storage, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the text similarity calculation method according to any one of claims 1 to 6.

Images