Text similarity calculation method and device based on Bert model
Through the text similarity calculation method based on the Bert model, combined with word frequency word presentation rate and Bert model characteristics, text matrix splicing and optimization are performed, which solves the problem of low accuracy in text similarity calculation in the existing technology, and achieves higher text Similarity calculation accuracy.
Patent Information
- Application Number
- CN202010151330.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-06
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-03-06
AI Technical Summary
The existing text similarity calculation methods have insufficient semantic recognition ability and word correlation, resulting in low accuracy.
Using the text similarity calculation method based on the Bert model, the first text matrix and the second text matrix were obtained through the word frequency word presentation algorithm and the pre-trained Bert model, and then the combination was spliced and feature optimization was performed, using the preset similarity. The algorithm calculates text similarity.
It improves the accuracy of text similarity calculation, comprehensively reflects the characteristics of word frequency, word presentation rate, relationship between words and sentences, and enhances the feature expression ability.
Smart Images

Figure CN111368037B_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for calculating text similarity based on a Bert model. [Background technology]
[0002] Text similarity calculation is one of the branches of natural language processing. At present, there are still problems in text similarity prediction and calculation, such as weak semantic recognition ability and weak correlation between words and text. The accuracy of text similarity calculation cannot meet users' expectations. [Summary of the invention]
[0003] In view of this, the embodiments of the present invention provide a method, apparatus, computer device and storage medium for calculating text similarity based on the Bert model, so as to solve the problem of low accuracy in current text similarity calculation.
[0004] In a first aspect, an embodiment of the present invention provides a method for calculating text similarity based on a Bert model, comprising:
[0005] Determine the text segments to be compared;
[0006] Using a word frequency and word appearance rate algorithm, a first text matrix is obtained based on the text segments to be compared;
[0007] Obtaining a second text matrix based on the text segments to be compared by using a pre-trained Bert model;
[0008] splicing the first text matrix and the second text matrix to obtain a spliced text matrix;
[0009] Performing feature optimization on the concatenated text matrix to obtain a target text matrix;
[0010] A preset similarity algorithm is used to obtain the text similarity between the text segments to be compared according to the target text matrix.
[0011] According to the above aspects and any possible implementation, an implementation is further provided, wherein the word frequency and word appearance rate algorithm is used to obtain a first text matrix based on the text segments to be compared, including:
[0012] Establishing a word bag according to the text segment to be compared, wherein the word bag includes words appearing in the text segment to be compared;
[0013] The frequency of each word in the text segment to be compared is calculated based on the word bag, and the formula is expressed as follows: Among them, t represents a word, d represents a sentence, tf t,dIndicates whether word t appears in sentence d. If so, it is 1, and if not, it is g. t,d Take 0, g t,d Indicates the proportion of a word in a text segment to be compared;
[0014] The inverse document frequency is calculated based on the word bag, and the formula is expressed as: Where N represents the total number of files, which are predetermined, df t Indicates that there are n files containing word t, where n is an integer greater than or equal to 0;
[0015] The first text matrix is obtained according to the word frequency and the inverse file frequency, and the elements in the first text matrix are expressed by the formula Calculated.
[0016] According to the above aspects and any possible implementation, an implementation is further provided, wherein before obtaining the second text matrix based on the to-be-compared text segment through the pre-trained Bert model, a training process of the Bert model is further included, including the following steps:
[0017] Get the original corpus;
[0018] Segmenting the original corpus at the character level;
[0019] Constructing sentence pairs according to the original corpus, wherein the sentence pairs include positive sample sentence pairs and negative sample sentence pairs, the positive sample sentence pairs have contextual relationships between the sentences, and the negative sample sentence pairs do not have contextual relationships between the sentences;
[0020] Connecting the sentence pairs based on the original corpus after character-level segmentation;
[0021] Randomly masking 10 percent of the characters in the sentence pairs to obtain training corpus;
[0022] The training corpus is input into an initial Bert model for training to obtain the Bert model.
[0023] According to the above aspect and any possible implementation manner, an implementation manner is further provided, wherein the first text matrix and the second text matrix are spliced to obtain a spliced text matrix, including:
[0024] Determining whether the matrix sizes of the first text matrix and the second text matrix are the same;
[0025] If they are the same, concatenating the first text matrix and the second text matrix to obtain the concatenated text matrix;
[0026] If they are not the same, the principal component analysis method is used to reduce the dimension of the first text matrix so that the matrix size of the first text matrix is equal to the matrix size of the second text matrix, and the first file matrix and the second text matrix are spliced after the dimension reduction to obtain the spliced text matrix.
[0027] According to the above aspects and any possible implementation, an implementation is further provided, wherein the step of performing feature optimization on the concatenated text matrix to obtain a target text matrix includes:
[0028] Based on the principal component analysis method, the word vector v of the concatenated text matrix is calculated s , expressed as Where S is the concatenated text matrix, v t is the vector of word t in the concatenated text matrix, α is the preset smoothing parameter, and p t is the probability of a word appearing in a document;
[0029] The word vector v is obtained by using the truncated singular value decomposition method s The principal component u of
[0030] According to the word vector v s And the principal component u to the word vector v s Perform feature optimization to obtain the updated word vector, which is expressed as v′ s =v s -u(u T )v s , where T represents the transpose matrix operation;
[0031] According to the updated word vector v s 'Get the target text matrix.
[0032] According to the above aspects and any possible implementation, an implementation is further provided, wherein the method of using a preset similarity algorithm to obtain the text similarity between the text segments to be compared according to the target text matrix includes:
[0033] According to the updated word vector v in the text segment to be compared s ', obtain the target text vector corresponding to the text segment to be compared from the target text matrix;
[0034] The similarity between the target text vectors is calculated using a pre-similarity algorithm to obtain the similarity between the text segments to be compared, which is expressed as Wherein, q represents the target text vector of the first text segment to be compared, c represents the target text vector of the second text segment to be compared, and the text segments to be compared include the first text segment to be compared and the second text segment to be compared.
[0035] In a second aspect, an embodiment of the present invention provides a text similarity calculation device based on the Bert model, comprising:
[0036] A text determination module, used to determine the text segment to be compared;
[0037] A first acquisition module, configured to obtain a first text matrix based on the text segment to be compared by adopting a word frequency and word appearance rate algorithm;
[0038] A second acquisition module, configured to obtain a second text matrix based on the text segment to be compared by using a pre-trained Bert model;
[0039] A spliced text matrix acquisition module, used for splicing the first text matrix and the second text matrix to obtain a spliced text matrix;
[0040] A target text matrix acquisition module is used to perform feature optimization on the concatenated text matrix to obtain a target text matrix;
[0041] The text similarity calculation module is used to obtain the text similarity between the text segments to be compared according to the target text matrix using a preset similarity algorithm.
[0042] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for calculating text similarity based on the Bert model when executing the computer program.
[0043] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, comprising: a computer program, which, when executed by a processor, implements the steps of the above-mentioned text similarity calculation method based on the Bert model.
[0044] In an embodiment of the present invention, firstly, based on the text segments to be compared, a word frequency and word occurrence rate algorithm and a pre-trained Bert model are used to obtain a first text matrix and a second text matrix respectively, wherein the first text matrix can reflect the word frequency and word occurrence rate in the text segments to be compared, and the second text matrix can reflect the relationship between words, the relationship between word positions and the relationship between sentences in the text segments to be compared; then, the first text matrix and the second text matrix are spliced to obtain a spliced text matrix, and the features reflected by the first text matrix and the second text matrix are combined to form a comprehensive matrix reflecting the features in multiple aspects. It helps to improve the accuracy of text similarity in the future; then the concatenated text matrix is feature optimized to obtain the target text matrix, so that the concatenated matrix can remove some noise data through feature optimization and improve the feature expression ability of the target text matrix; finally, the preset similarity algorithm is used to obtain the text similarity between the text segments to be compared according to the target text matrix, and the calculation is based on the word frequency and word occurrence rate of the words in the text segments to be compared reflected in the target text matrix, as well as the relationship between words in the text segments to be compared, the word position relationship and the relationship between sentences, etc., which can effectively improve the accuracy of text similarity.
Brief Description of the Drawings
[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0046] Figure 1 is a flow chart of a method for calculating text similarity based on the Bert model in one embodiment of the present invention;
[0047] Figure 2 It is a principle block diagram of a text similarity calculation device based on the Bert model in one embodiment of the present invention;
[0048] Figure 3 is a schematic diagram of a computer device according to an embodiment of the present invention. [Specific implementation method]
[0049] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0050] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.
[0052] It should be understood that the term "and / or" used in this article is only a description of the same field of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0053] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.
[0054] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.
[0055] Figure 1 A flow chart of the text similarity calculation method based on the Bert model in this embodiment is shown. The text similarity calculation method based on the Bert model can be applied to a text similarity calculation system, and the text similarity calculation system can be used to implement the text similarity calculation between sentences. The text similarity calculation system can be specifically applied to a computer device, wherein the computer device is a device that can perform human-computer interaction with a user, including but not limited to computers, smart phones, tablets and other devices. Figure 1 As shown, the text similarity calculation method based on the Bert model includes:
[0056] S10: Determine the text segment to be compared.
[0057] The to-be-compared text segments represent segments used for text similarity comparison, and there may be multiple segments. Each time a text similarity comparison is performed, two segments are selected from the to-be-compared text segments for comparison.
[0058] It can be understood that if one wants to compare any two sentences in a novel, the text segments to be compared can be all the fields contained in the novel. In this case, all the segments in the novel are text segments to be compared.
[0059] S20: Using a word frequency and word appearance rate algorithm, a first text matrix is obtained based on the text segments to be compared.
[0060] Among them, the word frequency and word appearance rate algorithm is an algorithm for calculating the frequency of word appearance and the reverse file frequency (word appearance rate), which can reflect the importance of words in a sentence segment. In one embodiment, the word frequency and word appearance rate of each word in the text segment to be compared are calculated using the word frequency and word appearance rate algorithm, and a first text matrix is constructed to reflect the word frequency and word appearance rate characteristics of these words in the form of a matrix.
[0061] Furthermore, in step S20, a word frequency and word appearance rate algorithm is used to obtain a first text matrix based on the text segments to be compared, which specifically includes:
[0062] S21: Building a word bag according to the text segment to be compared, wherein the word bag includes words appearing in the text segment to be compared.
[0063] It can be understood that a bag of words refers to a collection of words.
[0064] In one embodiment, a bag of words corresponding to the text segment to be compared is established, and the word frequency and word occurrence rate of the words in the text segment to be compared can be calculated.
[0065] S22: Calculate the frequency of each word in the text segment to be compared based on the bag of words, using the formula: Among them, t represents a word, d represents a sentence, tf t,d Indicates whether word t appears in sentence d. If so, it is 1, and if not, it is g. t,d Take 0, g t,d Indicates the proportion of a word in a text segment to be compared.
[0066] In one embodiment, the word frequency of each word in the text segment to be compared is calculated by using the bag of words. Specifically, if the text segment A to be compared includes word a, word b, word c and word d, and the text segment B to be compared includes word a, word b, word e and word f, then g A,a =1, g A,c =1, g B,a =1, g B,c =0.
[0067] In this embodiment, by calculating the frequency of each word in the text segment to be compared, the proportion of a word in a text segment to be compared can be highlighted.
[0068] S23: Calculate the inverse document frequency based on the bag of words, using the formula: Where N represents the total number of files, the files are predetermined, df t It means there are n files containing word t, where n is an integer greater than or equal to 0.
[0069] The file here is predetermined, for example, a chapter of a novel can be stored in a file. In one embodiment, the reverse file frequency is calculated based on the bag of words, which can reflect the proportion of words in the file dimension.
[0070] S24: A first text matrix is obtained according to the word frequency and the inverse document frequency. The elements in the first text matrix are calculated using the formula Calculated.
[0071] Specifically, the first text matrix can be expressed as Formula w t,d The calculation is for each specific element in the first text matrix, where v ij Represents the i-th word in the j-th text segment to be compared.
[0072] It is understandable that each specific element in the first text matrix can be calculated by multiplying the word frequency and the word occurrence rate (inverse document frequency). In this embodiment, the importance of words is reflected by the word frequency and word occurrence rate by using the first text matrix.
[0073] Among them, word frequency reflects the frequency of occurrence in a sentence, while word occurrence rate mainly reflects the frequency of occurrence in a file, in order to prevent a word from appearing in large quantities in a file but in small quantities in other files. Multiplying word frequency and inverse file frequency can combine the word characteristics reflected by both, and more accurately reflect the importance of words (which can be understood as giving different weights to words).
[0074] In steps S21-S24, by adopting the word frequency and word occurrence rate algorithm, a first text matrix is obtained based on the text segments to be compared. In the first text matrix, the importance of words in the text segments to be compared is reflected by word frequency and word occurrence rate, which helps to improve the accuracy of subsequent calculation of text similarity.
[0075] S30: Obtain a second text matrix based on the text segments to be compared using the pre-trained Bert model.
[0076] In one embodiment, the text segments to be compared are imported into a pre-trained Bert model, and a second text matrix is output. The second text matrix can dynamically express the relationship between words, word position relationship and sentence relationship in the text segments to be compared with the help of the feature extraction characteristics of the Bert model, and reflect the characteristics of the text segments to be compared from multiple aspects such as words and sentences, which can improve the accuracy of subsequent calculation of text similarity.
[0077] Specifically, the present embodiment uses an initial Bert model with 24 training network layers and 1024 network nodes, wherein the training network layer includes Trm, which is a bidirectional Transformer encoder corresponding to each network node. n As the input layer of the training model, T1, T2…T n As the output layer of the training model, E1, E2…E n The connection relationship with Trm is fully connected.
[0078] It can be seen that the structure of the Bert model itself is not complicated, and the key point of Bert model training lies in the preprocessing of the corpus.
[0079] Furthermore, before step S30, that is, before obtaining the second text matrix based on the text segment to be compared by using the pre-trained Bert model, a Bert model training process is also included, and the Bert model training process specifically includes the following steps:
[0080] S31: Obtain original corpus.
[0081] Among them, the original corpus can be obtained from open source corpora.
[0082] Furthermore, the original corpus may include the corpus in the open source corpus and the same number of text segments to be compared as the corpus in the open source corpus, and the two respectively account for half of the original corpus. By adopting this method of obtaining the original corpus, the Bert model can have a strong feature extraction capability for the text segments to be compared while maintaining a certain generalization capability, which can improve the accuracy of feature extraction of the Bert model.
[0083] S32: Segment the original corpus at the character level.
[0084] Among them, character-level segmentation includes segmenting characters such as punctuation marks. This character-level segmentation is mainly used for subsequent connection of sentence pairs and removal of non-word characters such as punctuation marks.
[0085] S33: Construct sentence pairs according to the original corpus, wherein the sentence pairs include positive sample sentence pairs and negative sample sentence pairs. The positive sample sentence pairs have contextual relationships between the sentences, while the negative sample sentence pairs do not have contextual relationships between the sentences.
[0086] Here, a sentence pair refers to a group of sentences. In this embodiment, positive sample sentence pairs and negative sample sentence pairs are constructed according to whether there is a contextual relationship between sentences, so that the initial Bert model can learn the ability of the relationship between sentences when training the initial Bert model, wherein the initial Bert model refers to the Bert model using the initial weights.
[0087] S34: Connect sentence pairs based on the original corpus after character-level segmentation.
[0088] It can be understood that the positive and negative samples of sentence pairs constructed in step S33 are used to determine whether there is a contextual relationship between the sentence pairs. In this embodiment, when connecting sentence pairs, the characters are used as the smallest unit for connection, so that the initial Bert model can learn features with characters as the smallest unit during the training process. Specifically, the [SEP] label is used to connect sentences, [CLS] is used as the label at the beginning of the sentence, and [SEP] is used as the label at the end of the sentence. The use of labels to mark the position of the sentence itself and the front-and-back relationship between sentences can help the initial Bert model learn these features when training the initial Bert model.
[0089] S35: Randomly mask 10% of the characters in the sentence pairs to obtain training corpus.
[0090] Here, the mask mechanism of Bert model training is adopted, wherein the masked characters can be predicted by using the pre-trained prediction model. The random masking of characters can enable the model to make judgments during the training process, so that the trained model has stronger generalization ability and stronger feature extraction ability.
[0091] S36: Input the training corpus into the initial Bert model for training to obtain the Bert model.
[0092] In steps S31-S36, a specific embodiment of training the Bert model is provided. It can be seen that the key point of training the Bert model in this embodiment is the preprocessing of the corpus, the training corpus obtained by the preprocessing is input into the initial Bert model for training, and the initial weights in the initial Bert model are updated to obtain the Bert model.
[0093] S40: splicing the first text matrix and the second text matrix to obtain a spliced text matrix.
[0094] Here, splicing refers to the operation of matrix splicing.
[0095] In one embodiment, a concatenated text matrix obtained by concatenating the first text matrix and the second text matrix includes the features of the first text matrix and the second text matrix, and can reflect features such as word frequency and word occurrence in the text segments to be compared, the relationship between words, word position relationship, and relationship between sentences in the text segments to be compared.
[0096] Furthermore, in step S40, the first text matrix and the second text matrix are concatenated to obtain a concatenated text matrix, which specifically includes:
[0097] S41: Determine whether the matrix sizes of the first text matrix and the second text matrix are the same.
[0098] S42: If they are the same, concatenate the first text matrix and the second text matrix to obtain a concatenated text matrix.
[0099] S43: If they are not the same, the principal component analysis method is used to reduce the dimension of the first text matrix so that the matrix size of the first text matrix is equal to the matrix size of the second text matrix, and the first document matrix and the second text matrix are spliced after the dimension reduction to obtain a spliced text matrix.
[0100] It can be understood that the output of the Bert model can set different dimensions of output elements according to task requirements. Generally speaking, the first text matrix generated by the word frequency and word appearance rate algorithm is a sparse matrix, which can be reduced to the same dimension size as the second text matrix output by the Bert model by the principal component analysis method, wherein the principal component analysis method is the same as the method of feature optimization using principal component analysis in steps S51-S54, and the principal component analysis method used in steps S51-S54 can be referred to.
[0101] In steps S41-S43, the first text matrix and the second text matrix can be guaranteed to be identical in dimension, which is convenient for computer calculation and subsequent further processing.
[0102] S50: Perform feature optimization on the concatenated text matrix to obtain a target text matrix.
[0103] In one embodiment, feature optimization refers to reducing the dimension of the concatenated text matrix to remove noise data in the concatenated text matrix, which helps to improve the accuracy of text similarity calculation.
[0104] Furthermore, in step S50, feature optimization is performed on the concatenated text matrix to obtain a target text matrix, which specifically includes:
[0105] S51: Based on the principal component analysis method, calculate the word vector v of the concatenated text matrix s, expressed as Among them, S is the concatenated text matrix, v t is the vector of word t in the concatenated text matrix, α is the preset smoothing parameter, and p t is the probability of a word appearing in a document.
[0106] Specifically, α can be 0.0001.
[0107] S52: Use truncated singular value decomposition method to obtain word vector v s The principal component u of .
[0108] S53: According to the word vector v s And the principal component u for the word vector v s Perform feature optimization to obtain the updated word vector, which is expressed as v s ′=v s -u(u T )v s , where T represents the transpose matrix operation.
[0109] In one embodiment, feature optimization uses a principal component analysis method to remove noise data in the word vector.
[0110] S54: According to the updated word vector v s ’Get the target text matrix.
[0111] Understandably, the word vector v obtained by s 'Build a target text matrix according to the text segments to be compared and the index numbers of the fields in the text segments to be compared.
[0112] In steps S51-S54, a specific embodiment of obtaining a target text matrix is provided, and the principal component analysis method is used to remove noise data, which helps to improve the accuracy of text similarity calculation.
[0113] S60: Using a preset similarity algorithm, the text similarity between the to-be-compared text segments is obtained according to the target text matrix.
[0114] Furthermore, in step S60, a preset similarity algorithm is used to obtain the text similarity between the text segments to be compared according to the target text matrix, specifically including:
[0115] S61: Based on the updated word vector v in the text segment to be compared s ′, obtain the target text vector corresponding to the text segment to be compared from the target text matrix.
[0116] It can be understood that a text segment to be compared includes different words, and a word vector v s′ corresponds to a word, then it is necessary to update the word vector v according to the words included in the text segment to be compared s ′, select the matrix elements related to the text segment to be compared from the target text matrix to obtain the target text vector corresponding to the text segment to be compared.
[0117] S62: Use the pre-similarity algorithm to calculate the similarity between the target text vectors to obtain the similarity between the text segments to be compared, which is expressed as: Wherein, q represents the target text vector of the first text segment to be compared, c represents the target text vector of the second text segment to be compared, and the text segments to be compared include the first text segment to be compared and the second text segment to be compared.
[0118] In one embodiment, the similarity algorithm specifically adopts cosine similarity, which can focus on measuring the similarity from the direction, highlight the similarity of the text, and improve the accuracy of the text similarity.
[0119] In an embodiment of the present invention, firstly, based on the text segments to be compared, a word frequency and word occurrence rate algorithm and a pre-trained Bert model are used to obtain a first text matrix and a second text matrix respectively, wherein the first text matrix can reflect the word frequency and word occurrence rate in the text segments to be compared, and the second text matrix can reflect the relationship between words, the relationship between word positions and the relationship between sentences in the text segments to be compared; then, the first text matrix and the second text matrix are spliced to obtain a spliced text matrix, and the features reflected by the first text matrix and the second text matrix are combined to form a comprehensive matrix reflecting the features in multiple aspects. It helps to improve the accuracy of text similarity in the future; then the concatenated text matrix is feature optimized to obtain the target text matrix, so that the concatenated matrix can remove some noise data through feature optimization and improve the feature expression ability of the target text matrix; finally, the preset similarity algorithm is used to obtain the text similarity between the text segments to be compared according to the target text matrix, and the calculation is based on the word frequency and word occurrence rate of the words in the text segments to be compared reflected in the target text matrix, as well as the relationship between words in the text segments to be compared, the word position relationship and the relationship between sentences, etc., which can effectively improve the accuracy of text similarity.
[0120] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0121] Based on the text similarity calculation method based on the Bert model provided in the embodiment, the embodiment of the present invention further provides a device embodiment for implementing each step and method in the above method embodiment.
[0122] Figure 2 The principle block diagram of the text similarity calculation device based on the Bert model corresponding to the text similarity calculation method based on the Bert model in the embodiment is shown. Figure 2 As shown, the text similarity calculation device based on the Bert model includes a text determination module 10, a first acquisition module 20, a second acquisition module 30, a spliced text matrix acquisition module 40, a target text matrix acquisition module 50 and a text similarity calculation module 60. Among them, the implementation functions of the text determination module 10, the first acquisition module 20, the second acquisition module 30, the spliced text matrix acquisition module 40, the target text matrix acquisition module 50 and the text similarity calculation module 60 correspond to the steps corresponding to the text similarity calculation method based on the Bert model in the embodiment. To avoid redundancy, this embodiment will not be described in detail one by one.
[0123] The text determination module 10 is used to determine the text segment to be compared.
[0124] The to-be-compared text segments represent segments used for text similarity comparison, and there may be multiple segments. Each time a text similarity comparison is performed, two segments are selected from the to-be-compared text segments for comparison.
[0125] It can be understood that if one wants to compare any two sentences in a novel, the text segments to be compared can be all the fields contained in the novel. In this case, all the segments in the novel are text segments to be compared.
[0126] The first acquisition module 20 is used to obtain a first text matrix based on the text segments to be compared by adopting a word frequency and word presence algorithm.
[0127] Among them, the word frequency and word appearance rate algorithm is an algorithm for calculating the frequency of word appearance and the reverse file frequency (word appearance rate), which can reflect the importance of words in a sentence segment. In one embodiment, the word frequency and word appearance rate of each word in the text segment to be compared are calculated using the word frequency and word appearance rate algorithm, and a first text matrix is constructed to reflect the word frequency and word appearance rate characteristics of these words in the form of a matrix.
[0128] The second acquisition module 30 is used to obtain a second text matrix based on the text segment to be compared by using a pre-trained Bert model.
[0129] In one embodiment, the text segments to be compared are imported into a pre-trained Bert model, and a second text matrix is output. The second text matrix can dynamically express the relationship between words, word position relationship and sentence relationship in the text segments to be compared with the help of the feature extraction characteristics of the Bert model, and reflect the characteristics of the text segments to be compared from multiple aspects such as words and sentences, which can improve the accuracy of subsequent calculation of text similarity.
[0130] Specifically, the present embodiment uses an initial Bert model with 24 training network layers and 1024 network nodes, wherein the training network layer includes Trm, which represents a bidirectional Transformer encoder corresponding to each network node. n As the input layer of the training model, T1, T2…T n As the output layer of the training model, E1, E2…E n The connection relationship with Trm is fully connected.
[0131] It can be seen that the structure of the Bert model itself is not complicated, and the key point of Bert model training lies in the preprocessing of the corpus.
[0132] The spliced text matrix acquisition module 40 is used to splice the first text matrix and the second text matrix to obtain a spliced text matrix.
[0133] Here, splicing refers to the operation of matrix splicing.
[0134] In one embodiment, a concatenated text matrix obtained by concatenating the first text matrix and the second text matrix includes the features of the first text matrix and the second text matrix, and can reflect features such as word frequency and word occurrence in the text segments to be compared, the relationship between words, word position relationship, and relationship between sentences in the text segments to be compared.
[0135] The target text matrix acquisition module 50 is used to perform feature optimization on the concatenated text matrix to obtain a target text matrix.
[0136] In one embodiment, feature optimization may specifically refer to reducing the dimension of the concatenated text matrix to remove noise data in the concatenated text matrix, which helps to improve the accuracy of text similarity calculation.
[0137] The text similarity calculation module 60 is used to obtain the text similarity between the text segments to be compared according to the target text matrix using a preset similarity algorithm.
[0138] Optionally, the first acquisition module 20 is specifically configured to:
[0139] Establish a word bag according to the text segment to be compared, wherein the word bag includes words appearing in the text segment to be compared;
[0140] The frequency of each word in the text segment to be compared is calculated based on the bag of words, and the formula is expressed as: Among them, t represents a word, d represents a sentence, tf t,d Indicates whether word t appears in sentence d. If so, it is 1, and if not, it is g. t,dTake 0, g t,d Indicates the proportion of a word in a text segment to be compared;
[0141] The inverse document frequency is calculated based on the bag of words, and the formula is expressed as Where N represents the total number of files, the files are predetermined, df t It means there are n files containing word t, where n is an integer greater than or equal to 0;
[0142] The first text matrix is obtained based on the word frequency and the inverse document frequency. The elements in the first text matrix are calculated using the formula Calculated.
[0143] Optionally, the text similarity calculation device based on the Bert model further includes a training module, which is used to:
[0144] Get the original corpus;
[0145] Segment the original corpus at the character level;
[0146] Construct sentence pairs according to the original corpus, wherein the sentence pairs include positive sample sentence pairs and negative sample sentence pairs, the positive sample sentence pairs have contextual relationships between the sentences, and the negative sample sentence pairs do not have contextual relationships between the sentences;
[0147] Connect sentence pairs based on the original corpus after character-level segmentation;
[0148] Randomly mask 10% of the characters in the sentence pairs to obtain the training corpus;
[0149] The training corpus is input into the initial Bert model for training to obtain the Bert model.
[0150] Optionally, the spliced text matrix acquisition module 40 is specifically used for:
[0151] Determining whether the matrix sizes of the first text matrix and the second text matrix are the same;
[0152] If they are the same, concatenate the first text matrix with the second text matrix to obtain a concatenated text matrix;
[0153] If they are not the same, the principal component analysis method is used to reduce the dimension of the first text matrix so that the matrix size of the first text matrix is equal to the matrix size of the second text matrix, and the first document matrix and the second text matrix are spliced after the dimension reduction to obtain a spliced text matrix.
[0154] Optionally, the target text matrix acquisition module 50 is specifically used for:
[0155] Based on the principal component analysis method, the word vector v of the concatenated text matrix is calculated. s , expressed as Among them, S is the concatenated text matrix, v t is the vector of word t in the concatenated text matrix, α is the preset smoothing parameter, and p t is the probability of a word appearing in a document;
[0156] The word vector v is obtained by truncated singular value decomposition method s The principal component u of
[0157] According to the word vector v s And the principal component u for the word vector v s Perform feature optimization to obtain the updated word vector, which is expressed as v s ′=v s -u(u T )v s , where T represents the transpose matrix operation;
[0158] According to the updated word vector v s ’Get the target text matrix.
[0159] Optionally, the text similarity calculation module 60 is specifically used for:
[0160] According to the updated word vector v in the text segment to be compared s ′, obtain the target text vector corresponding to the text segment to be compared from the target text matrix;
[0161] The similarity between target text vectors is calculated by using the pre-similarity algorithm to obtain the similarity between the text segments to be compared, which is expressed as Wherein, q represents the target text vector of the first text segment to be compared, c represents the target text vector of the second text segment to be compared, and the text segments to be compared include the first text segment to be compared and the second text segment to be compared.
[0162] In an embodiment of the present invention, firstly, based on the text segments to be compared, a word frequency and word occurrence rate algorithm and a pre-trained Bert model are used to obtain a first text matrix and a second text matrix respectively, wherein the first text matrix can reflect the word frequency and word occurrence rate in the text segments to be compared, and the second text matrix can reflect the relationship between words, the relationship between word positions and the relationship between sentences in the text segments to be compared; then, the first text matrix and the second text matrix are spliced to obtain a spliced text matrix, and the features reflected by the first text matrix and the second text matrix are combined to form a comprehensive matrix reflecting the features in multiple aspects. It helps to improve the accuracy of text similarity in the future; then the concatenated text matrix is feature optimized to obtain the target text matrix, so that the concatenated matrix can remove some noise data through feature optimization and improve the feature expression ability of the target text matrix; finally, the preset similarity algorithm is used to obtain the text similarity between the text segments to be compared according to the target text matrix, and the calculation is based on the word frequency and word occurrence rate of the words in the text segments to be compared reflected in the target text matrix, as well as the relationship between words in the text segments to be compared, the word position relationship and the relationship between sentences, etc., which can effectively improve the accuracy of text similarity.
[0163] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the text similarity calculation method based on the Bert model in the embodiment is implemented. To avoid repetition, it is not described here one by one. Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the text similarity calculation device based on the Bert model in the embodiment are implemented. To avoid repetition, it is not described here one by one.
[0164] Figure 3 Schematic diagram of a computer device provided by an embodiment of the present invention. Figure 3 As shown, the computer device 70 of this embodiment includes: a processor 71, a memory 72, and a computer program 73 stored in the memory 72 and executable on the processor 71. When the computer program 73 is executed by the processor 71, the text similarity calculation method based on the Bert model in the embodiment is implemented. Alternatively, when the computer program 73 is executed by the processor 71, the functions of each model / unit in the text similarity calculation device based on the Bert model corresponding to the text similarity calculation method based on the Bert model in the embodiment are implemented.
[0165] The computer device 70 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 70 may include, but is not limited to, a processor 71 and a memory 72. Those skilled in the art will appreciate that Figure 3This is only an example of the computer device 70 and does not constitute a limitation of the computer device 70. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.
[0166] The processor 71 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0167] The memory 72 may be an internal storage unit of the computer device 70, such as a hard disk or memory of the computer device 70. The memory 72 may also be an external storage device of the computer device 70, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 70. Further, the memory 72 may also include both an internal storage unit of the computer device 70 and an external storage device. The memory 72 is used to store computer programs and other programs and data required by the computer device. The memory 72 may also be used to temporarily store data that has been output or is to be output.
[0168] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0169] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A text similarity calculation method based on the Bert model, It is characterized in that include: Determine the text segments to be compared; Using a word frequency and word appearance rate algorithm, a first text matrix is obtained based on the text segments to be compared; Obtaining a second text matrix based on the to-be-compared text segment through a pre-trained Bert model, wherein the second text matrix is used to dynamically express the relationship between words, the relationship between word positions, and the relationship between sentences in the to-be-compared text segment; splicing the first text matrix and the second text matrix to obtain a spliced text matrix; Performing feature optimization on the concatenated text matrix to obtain a target text matrix; Using a preset similarity algorithm, the text similarity between the text segments to be compared is obtained according to the target text matrix; The method of using a word frequency and word appearance rate algorithm to obtain a first text matrix based on the text segments to be compared includes: Establishing a word bag according to the text segment to be compared, wherein the word bag includes words appearing in the text segment to be compared; The frequency of each word in the text segment to be compared is calculated based on the word bag, and the formula is expressed as follows: Among them, t represents a word, d represents a sentence, tf t,d Indicates whether word t appears in sentence d. If so, g t,d Take 1, if it does not appear, then g t,d Take 0, g t,d Indicates the proportion of a word in a text segment to be compared; The inverse document frequency is calculated based on the word bag, and the formula is expressed as: Where N represents the total number of files, which are predetermined, df t Indicates that there are n files containing word t, where n is an integer greater than or equal to 0; Obtaining the first text matrix according to the word frequency and the inverse document frequency, wherein elements in the first text matrix are calculated by multiplying the word frequency and the inverse document frequency; Before obtaining the second text matrix based on the to-be-compared text segment by using the pre-trained Bert model, a training process of the Bert model is also included, including the following steps: Get the original corpus; Segmenting the original corpus at the character level; Constructing sentence pairs according to the original corpus, wherein the sentence pairs include positive sample sentence pairs and negative sample sentence pairs, the positive sample sentence pairs have contextual relationships between sentences, and the negative sample sentence pairs do not have contextual relationships between sentences, and marking the positions of the sentences themselves and the preceding and following relationships between sentences through labels; Connecting the sentence pairs based on the original corpus after character-level segmentation; Randomly masking 10 percent of the characters in the sentence pairs to obtain training corpus; Inputting the training corpus into an initial Bert model for training to obtain the Bert model; The step of performing feature optimization on the concatenated text matrix to obtain a target text matrix includes: Based on the principal component analysis method, the word vector v of the concatenated text matrix is calculated s , expressed as Where S is the concatenated text matrix, v t is the vector of word t in the concatenated text matrix, α is the preset smoothing parameter, and p t is the probability of word t appearing in the document; The word vector v is obtained by using the truncated singular value decomposition method s The principal component u of According to the word vector v s And the principal component u to the word vector v s Perform feature optimization to obtain the updated word vector, which is expressed as v′ s =v s -u(u T )v s , where T represents the transpose matrix operation; According to the updated word vector v′ s The target text matrix is obtained.
2. The method according to claim 1, It is characterized in that The step of splicing the first text matrix and the second text matrix to obtain a spliced text matrix includes: Determining whether the matrix sizes of the first text matrix and the second text matrix are the same; If they are the same, concatenating the first text matrix and the second text matrix to obtain the concatenated text matrix; If they are not the same, the principal component analysis method is used to reduce the dimension of the first text matrix so that the matrix size of the first text matrix is equal to the matrix size of the second text matrix, and the first text matrix and the second text matrix are spliced after the dimension reduction to obtain the spliced text matrix.
3. The method according to claim 1, It is characterized in that The method of using a preset similarity algorithm to obtain the text similarity between the text segments to be compared according to the target text matrix includes: According to the updated word vector v′ in the text segment to be compared s , obtaining a target text vector corresponding to the text segment to be compared from the target text matrix; The similarity between the target text vectors is calculated using a pre-similarity algorithm to obtain the similarity between the text segments to be compared, which is expressed as Wherein, q represents the target text vector of the first text segment to be compared, c represents the target text vector of the second text segment to be compared, and the text segments to be compared include the first text segment to be compared and the second text segment to be compared.
4. A text similarity calculation device based on the Bert model, It is characterized in that The device comprises: A text determination module, used to determine the text segment to be compared; A first acquisition module, configured to obtain a first text matrix based on the text segment to be compared by adopting a word frequency and word appearance rate algorithm; A second acquisition module, configured to obtain a second text matrix based on the text segment to be compared by using a pre-trained Bert model, wherein the second text matrix is used to dynamically express the relationship between words, the relationship between word positions, and the relationship between sentences in the text segment to be compared; A spliced text matrix acquisition module, used for splicing the first text matrix and the second text matrix to obtain a spliced text matrix; A target text matrix acquisition module is used to perform feature optimization on the concatenated text matrix to obtain a target text matrix; A text similarity calculation module, used to obtain the text similarity between the text segments to be compared according to the target text matrix by using a preset similarity algorithm; The first acquisition module is specifically used to: Establishing a word bag according to the text segment to be compared, wherein the word bag includes words appearing in the text segment to be compared; The frequency of each word in the text segment to be compared is calculated based on the word bag, and the formula is expressed as follows: Among them, t represents a word, d represents a sentence, tf t,d Indicates whether word t appears in sentence d. If so, g t,d Take 1, if it does not appear, then g t,d Take 0, g t,d Indicates the proportion of a word in a text segment to be compared; The inverse document frequency is calculated based on the word bag, and the formula is expressed as: Where N represents the total number of files, which are predetermined, df t Indicates that there are n files containing word t, where n is an integer greater than or equal to 0; Obtaining the first text matrix according to the word frequency and the inverse document frequency, wherein elements in the first text matrix are calculated by multiplying the word frequency and the inverse document frequency; Before obtaining the second text matrix based on the to-be-compared text segment by using the pre-trained Bert model, a training process of the Bert model is also included, including the following steps: Get the original corpus; Segmenting the original corpus at the character level; Constructing sentence pairs according to the original corpus, wherein the sentence pairs include positive sample sentence pairs and negative sample sentence pairs, the positive sample sentence pairs have contextual relationships between sentences, and the negative sample sentence pairs do not have contextual relationships between sentences, and marking the positions of the sentences themselves and the preceding and following relationships between sentences through labels; Connecting the sentence pairs based on the original corpus after character-level segmentation; Randomly masking 10 percent of the characters in the sentence pairs to obtain training corpus; Inputting the training corpus into an initial Bert model for training to obtain the Bert model; The step of performing feature optimization on the concatenated text matrix to obtain a target text matrix includes: Based on the principal component analysis method, the word vector v of the concatenated text matrix is calculated s , expressed as Where S is the concatenated text matrix, v t is the vector of word t in the concatenated text matrix, α is the preset smoothing parameter, and p t is the probability of word t appearing in the document; The word vector v is obtained by using the truncated singular value decomposition method s The principal component u of According to the word vector v s And the principal component u to the word vector v s Perform feature optimization to obtain the updated word vector, which is expressed as v′ s =v s -u(u T )v s , where T represents the transpose matrix operation; According to the updated word vector v′ s The target text matrix is obtained.
5. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method for calculating text similarity based on the Bert model as described in any one of claims 1 to 3 are implemented.
6. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method for calculating text similarity based on the Bert model as claimed in any one of claims 1 to 3 are implemented.
