A text similarity analysis method and system based on residual fusion and stacked integration

Abstract

The application discloses a text similarity analysis method and system based on residual fusion and stacked integration. The method comprises the following steps: inputting the text to be analyzed into a target text similarity judgment model for similarity judgment to obtain a judgment result; the target text similarity judgment model is obtained by fusing a representative SBERT model and an interactive ABCNN model; if the text to be analyzed is judged as similar text, inputting the text to be analyzed into a target text similarity calculation model for similarity calculation to obtain a target text similarity value; the target text similarity calculation model is obtained by parallel heterogeneous integration of multiple different interactive pre-training models. The technical problem that current text similarity analysis models are difficult to have high retrieval efficiency and accuracy is solved, accurate text similarity judgment and high-accuracy text similarity calculation are realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of text similarity analysis, and more particularly, relates to a text similarity analysis method and system based on residual fusion and stacked integration. BACKGROUND

[0002] Text similarity analysis, also known as "semantic similarity matching", is one of the most basic and core tasks in the field of natural language processing. The core of practical applications such as search engines, question and answer systems, and paper plagiarism detection is to realize text similarity analysis based on respective corpora. The essence of text similarity analysis is to judge the similar categories to which the vectorized text belongs, or to calculate the spatial distance between the text feature vectors. Therefore, text similarity analysis is divided into text similarity judgment and text similarity calculation. Among them, text similarity judgment is used for qualitative analysis of whether two texts are semantically similar, which belongs to the classification task. Text similarity calculation is a quantitative calculation of the semantic similarity between two texts, which belongs to the regression task, and is a quantitative calculation of the semantic correlation between two texts that the text similarity judgment model cannot provide. The text similarity calculation model can also realize text similarity judgment based on the similarity threshold set by humans or programs automatically. However, the self-defined process of the text similarity threshold is complex and tedious and is affected by subjective factors, so the mainstream implementation method of the text similarity judgment task is still the text similarity judgment model.

[0003] Currently, text similarity analysis is realized based on a text similarity analysis model, and the text similarity analysis model is divided into interactive and representative types, each of which has its own advantages and disadvantages. Among them, the interactive text similarity analysis model based on a complex attention mechanism allows the text to fully interact with the semantics, has a high accuracy, but has low text retrieval efficiency. The representative text similarity analysis model usually directly calculates the cosine similarity of two text vectors or performs classification operations, which cannot fully compare the semantics of two texts, has a low accuracy, but has a high text retrieval efficiency due to supporting local storage of feature vectors.

[0004] However, the respective advantages and disadvantages of the interactive and representative text similarity analysis models, the current mainstream practice in the industry is to use the interactive model to pursue accuracy, and to use the representative model to pursue text retrieval efficiency. However, the semantic defocusing problem of the representative model and the low retrieval efficiency problem of the interactive model make it difficult for the current text similarity analysis model to have high retrieval efficiency and accuracy. SUMMARY

[0005] In view of defects of the prior art, the present application aims to provide a text similarity analysis method and system based on residual fusion and stacked integration, aiming to solve the problem that the current text similarity analysis model is difficult to have high retrieval efficiency and accuracy due to low efficiency of an interactive text similarity analysis model and low accuracy of a representation text similarity analysis model.

[0006] To achieve the above-mentioned purpose, the present application provides a text similarity analysis method based on residual fusion and stacked integration, comprising:

[0007] inputting the text to be analyzed into a target text similarity judgment model for similarity judgment to obtain a judgment result; the target text similarity judgment model is obtained by fusing a representation SBERT model and an interactive ABCNN model;

[0008] if the text to be analyzed is judged as similar text, inputting the text to be analyzed into a target text similarity calculation model for similarity calculation to obtain a target text similarity value; the target text similarity calculation model is obtained by parallel heterogeneous integration of multiple different interactive pre-training models.

[0009] Optionally, the target text similarity judgment model is obtained by fusing the representation SBERT model and the interactive ABCNN model, comprising:

[0010] fusing the representation SBERT model and the interactive ABCNN model by using an identity mapping method of a residual network to obtain an initial text similarity judgment model;

[0011] iteratively training the initial text similarity judgment model based on a transfer learning method to obtain a target text similarity judgment model.

[0012] Optionally, the fusing the representation SBERT model and the interactive ABCNN model by using the identity mapping method of the residual network to obtain the text similarity judgment model, comprising:

[0013] constructing a text similarity judgment model, comprising an input layer, an embedding layer, a feature extraction layer, a feature interaction layer, a fusion layer and an output layer;

[0014] the input layer pre-processes the training text and manually labels the training text based on a non-type;

[0015] the embedding layer vectorizes the labeled training text and generates vector features of the training text through a Mengzi tokenizer;

[0016] The feature extraction layer adopts an SBERT model of a representation type, and a BERT pre-training model in the SBERT model is replaced with a lightweight Mengzi pre-training model; the vector feature is provided as standard input to the Mengzi pre-training model to obtain a fixed-dimension dense feature vector;

[0017] The feature interaction layer adopts an ABCNN model of an interaction type, and the feature interaction layer fully interacts the features extracted by the SBERT model according to an attention mechanism to extract deep second semantic feature vectors o1 and o2;

[0018] The fusion layer splices the first semantic feature vectors u1 and u2 input by the feature extraction layer and the second semantic feature vectors o1 and o2 input by the feature interaction layer in a feature dimension layer to obtain a vector after semantic feature fusion;

[0019] The output layer includes sequentially connected full connection layers, a normalization layer, an activation function, a full connection layer and a softmax function, and obtains a text similarity judgment result according to the vector after semantic feature fusion.

[0020] Optionally, the vector feature is provided as standard input to the Mengzi pre-training model to obtain a fixed-dimension dense feature vector, including:

[0021] After the vector feature is provided as standard input to the Mengzi pre-training model, first semantic feature vectors u1 and u2 with a [CLS] identifier and other semantic vectors v1 and v2 formed by splicing features of each word are calculated through formulas (1) and (2); the formulas are as follows:

[0022] u1 = last_hidden_state[:, 0] (1)

[0023] v1 = last_hidden_state[:, 1:-1] (2)

[0024] The other semantic vectors v1 and v2 are output to the feature interaction layer for implicit semantic extraction, and the first semantic feature vectors u1 and u2 are output to the fusion layer.

[0025] Optionally, the initial text similarity judgment model is iteratively trained based on a transfer learning method to obtain a target text similarity judgment model, including:

[0026] The source domain model is iteratively trained based on a “yes-no type” artificial annotation training set, and the trained model is stored in a local device in the form of an ordered dictionary state_dict;

[0027] Traverse the network model parameters of the source domain model and the target domain model, find and record the common parameters of the two; if the source domain model and the target domain model have common parameters, use the parameter values stored in the local source domain model to initialize the same parameters in the target domain model;

[0028] After the target domain model parameter initialization is completed, the Mengzi pre-training model is trained and optimized using the CoSENT loss function; wherein the CoSENT loss function is expressed as:

[0029]

[0030] The text similarity judgment always satisfies the constraint condition during the iterative training process, that is, the similarity between positive samples u i and u j is always greater than the similarity between negative samples u k and u l ; the constraint condition is shown in formula (4):

[0031] cos(u i ,u j )>cos(u k ,u l ) (4)

[0032] The text editing-based data enhancement method quickly multiplies the rare samples in the training data set, and the data enhancement method includes any one or more of synonym random insertion, synonym random replacement, arbitrary word random exchange and arbitrary word random deletion.

[0033] Optionally, the target text similarity calculation model is obtained by parallel heterogeneous integration of multiple different interactive pre-training models, including:

[0034] The Roberta-large, Roberta-pair, ERNIE and Bert-wwm-ext pre-training models are parallel heterogeneous integrated into a double-layer structure model by using a stacking integration method to obtain an initial text similarity calculation model;

[0035] The initial text similarity calculation model is iteratively trained based on the adversarial training and FP16 mixed precision method to obtain the target text similarity calculation model.

[0036] Optionally, the Roberta-large, Roberta-pair, ERNIE and Bert-wwm-ext pre-training models are parallel heterogeneous integrated into a double-layer structure model by using a stacking integration method to obtain an initial text similarity calculation model, including:

[0037] The original training set is divided into three equal parts, which are provided to three primary learners Roberta-large, Roberta-pair and ERNIE pre-training models for training;

[0038] The three primary learners use 5-fold cross-validation method to calculate three first text similarity values respectively, and the three first text similarity values are horizontally spliced to generate a secondary training set;

[0039] The secondary learner Bert-wwm-ext pre-training model uses 5-fold cross-validation method to divide the secondary training set into five equal parts, one of which is used as a validation set and the rest as a training set, which is provided to the Bert-wwm-ext pre-training model for training. The model with the best performance is saved locally;

[0040] The stored BERT-wwm-ext model is used to calculate the text similarity on the secondary training set to obtain a second text similarity value. The first text similarity value and the second text similarity value are weighted and averaged to obtain a target text similarity value.

[0041] Optionally, the initial text similarity calculation model is iteratively trained based on the adversarial training and FP16 mixed precision method to obtain a target text similarity calculation model, comprising:

[0042] The FGM adversarial training and FP16 mixed precision method are used to realize the iterative training of the text similarity calculation model. The adversarial training method is shown in formula (5):

[0043]

[0044] The feature vectors of each learner before normalization are used for text similarity calculation using the inverse function logit of sigmoid, and the calculated text similarity is arithmetically averaged, geometrically averaged and harmonically averaged. The inverse function logit is shown in formula (6):

[0045]

[0046] Optionally, it further comprises:

[0047] The positive and negative samples are proportionally sampled to supplement the secondary training set;

[0048] The proportionally sampled positive samples are represented as:

[0049] num pos =floor(len(temp falses )×rate origin ) (7)

[0050] The proportional sampling of negative samples is expressed as:

[0051] num neg = floor(len(temp trues ) ÷ rate origin ) (8)

[0052] Wherein, num pos is the sampling number of positive samples, num neg is the sampling number of negative samples, temp falses and temp trues are lists respectively storing new positive and negative samples, rate origin is the ratio of the original number of positive and negative samples.

[0053] In the second aspect, the application further provides a text similarity analysis system based on residual fusion and stacked ensemble, comprising:

[0054] A text similarity judgment module is configured to input the text to be analyzed into a target text similarity judgment model to perform similarity judgment and obtain a judgment result; the target text similarity judgment model is obtained by fusing an SBERT model of a representation type and an ABCNN model of an interaction type;

[0055] A text similarity calculation module is configured to input the text to be analyzed into a target text similarity calculation model to perform similarity calculation and obtain a target text similarity value if the text to be analyzed is judged as a similar text; the target text similarity calculation model is obtained by parallel heterogeneous ensemble of multiple different interaction type pre-training models.

[0056] Compared with the prior art, the above technical scheme of the application can achieve the following beneficial effects:

[0057] 1. The target text similarity judgment model provided by the application fuses the semantic features of the representation type model and the interaction type model based on the identity mapping method of the residual network, realizes iterative training of the model based on the transfer learning method, and optimizes the accuracy and robustness of the model by fine-tuning the pre-training model, the loss function based on the cosine value, and the data enhancement method based on text editing.

[0058] 2. The target text similarity calculation model provided by the application realizes hierarchical heterogeneous ensemble of multiple interaction type pre-training models with different performance based on the stacked ensemble method, realizes iterative training of the model based on the adversarial training and FP16 mixed precision method, and introduces the text similarity calculation method based on the sigmoid inverse function and the data enhancement method based on the text similarity transmission to realize the accuracy and robustness optimization of the model.

[0059] 3、The text similarity analysis method based on residual fusion and stacked integration provided by the application utilizes a target text similarity judgment model and a target text similarity calculation model to perform similarity judgment and similarity calculation on the text to be analyzed, realizes text similarity judgment with both accuracy and retrieval efficiency, and high-accuracy text similarity calculation. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of a text similarity analysis method based on residual fusion and stacked integration provided by the first embodiment of the application.

[0061] Figure 2 is a structural diagram of a target text similarity judgment model provided by the first embodiment of the application.

[0062] Figure 3 is a structural diagram of a target text similarity calculation model provided by the first embodiment of the application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the application clearer, the application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0064] The content involved in the above embodiments is described below in combination with a preferred embodiment.

[0065] Embodiment One

[0066] Figure 1 is a flowchart of a text similarity analysis method based on residual fusion and stacked integration provided by the first embodiment of the application.

[0067] As shown in Figure 1 , a text similarity analysis method based on residual fusion and stacked integration includes:

[0068] S1, input the text to be analyzed into a target text similarity judgment model for similarity judgment to obtain a judgment result; the target text similarity judgment model is obtained by fusing an SBERT model of a representation type and an ABCNN model of an interactive type.

[0069] S2, if the text to be analyzed is judged as similar text, input the text to be analyzed into a target text similarity calculation model for similarity calculation to obtain a target text similarity value; the target text similarity calculation model is obtained by parallel heterogeneous integration of multiple different interactive pre-training models.

[0070] The target text similarity judgment model is based on an identity mapping method of a residual network to realize semantic feature fusion of a representation type model and an interaction type model, and is based on a transfer learning method to realize iterative training of the model, and realizes accuracy and robustness optimization of the model by fine-tuning a pre-trained model, a loss function based on a cosine value, and a data enhancement method based on text editing.

[0071] The target text similarity judgment model is based on an identity mapping method of a residual network to realize semantic feature fusion of a representation type model and an interaction type model, and is based on a transfer learning method to realize iterative training of the model, and realizes accuracy and robustness optimization of the model by fine-tuning a pre-trained model, a loss function based on a cosine value, and a data enhancement method based on text editing.

[0072] The target text similarity judgment model is based on an identity mapping method of a residual network to realize semantic feature fusion of a representation type model and an interaction type model, and is based on a transfer learning method to realize iterative training of the model, and realizes accuracy and robustness optimization of the model by fine-tuning a pre-trained model, a loss function based on a cosine value, and a data enhancement method based on text editing.

[0073] Optionally, the target text similarity judgment model is obtained by fusing a representation type SBERT model and an interaction type ABCNN model, and includes:

[0074] An identity mapping method of a residual network is used to fuse the representation type SBERT model and the interaction type ABCNN model to obtain an initial text similarity judgment model.

[0075] The initial text similarity judgment model is iteratively trained based on a transfer learning method to obtain the target text similarity judgment model.

[0076] The overall structure of the target text similarity judgment model is as shown in Figure 2 The identity mapping method of the residual network fuses the improved representation type SBERT model and the interaction type ABCNN model using the following formula,

[0077] H(x)=F(x)+x

[0078] When the interaction type model does not capture any text semantic features, i.e. F(x)=0, the text similarity judgment fusion model captures text semantic features at least equal to the semantic features captured by the representation type model that constitutes it, i.e. H(x)=x, thereby realizing semantic information identity mapping at the feature level, and further ensuring that the performance of the initial text similarity judgment model is at least not worse than that of the representation type model that constitutes it.

[0079] The above initial text similarity judgment model is iteratively trained using a model-based transfer learning method, which includes iterative training of a source domain model and parameter initialization of a target domain model.

[0080] Optionally, the identity mapping method using the residual network fuses the representation type SBERT model and the interaction type ABCNN model to obtain a text similarity judgment model, comprising:

[0081] The text similarity judgment model is constructed and comprises an input layer, an embedding layer, a feature extraction layer, a feature interaction layer, a fusion layer and an output layer.

[0082] The input layer pre-processes the training text and manually labels the training text based on the yes-no type.

[0083] The embedding layer vectorizes the labeled training text and generates vector features of the training text through the Mengzi tokenizer.

[0084] The feature extraction layer uses the representation type SBERT model, and replaces the BERT pre-training model in the SBERT model with a lightweight Mengzi pre-training model; the vector features are provided as standard input to the Mengzi pre-training model to obtain a fixed-dimension dense feature vector.

[0085] The feature interaction layer uses the interaction type ABCNN model, and fully interacts the features extracted by the SBERT model according to the attention mechanism to extract deep second semantic feature vectors o1 and o2.

[0086] The fusion layer splices the first semantic feature vectors u1 and u2 input by the feature extraction layer and the second semantic feature vectors o1 and o2 input by the feature interaction layer in the feature dimension layer based on the vector splicing method to obtain a vector after semantic feature fusion.

[0087] The output layer comprises sequentially connected fully connected layers, a normalization layer, an activation function, a fully connected layer and a softmax function, and obtains a text similarity judgment result according to the vector after semantic feature fusion.

[0088] From Figure 2 As can be seen, the structure of the initial text similarity judgment model is divided into six levels according to function, which are: input layer, embedding layer, feature extraction layer, feature interaction layer, fusion layer and output layer. Among them, the feature extraction layer, the feature interaction layer and the fusion layer are the core modules of the model. The specific content of each layer in the overall structure of the initial text similarity judgment model is as follows:

[0089] (1) Input layer: text preprocessing based on yes-no type artificial labeling data in the form of (text1, text2, label).

[0090] (2) Embedding layer: The main function of this layer is to realize text vectorization and provide standard input for the Mengzi pre-trained model of the feature extraction layer. The structure of the Mengzi pre-trained model is consistent with BERT, and the industry usually uses the official BERT word segmenter directly to generate standard input for the BERT pre-trained model. Therefore, the output of the embedding layer is generated by the Mengzi word segmenter.

[0091] (3) Feature Extraction Layer: This layer is an improved SBERT model, replacing the BERT pre-trained model in SBERT with the lightweight Chinese pre-trained model Mengzi. The model structure of Mengzi is consistent with BERT, allowing for rapid replacement of the two. This layer maps the standard input provided by the embedding layer into a dense feature vector of fixed dimensions in the feature space, which is used to extract the lexical, syntactic and contextual semantic features of the input text.

[0092] Specifically, this includes:

[0093] After the vector features are provided as standard input to the Mengzi pre-trained model, the output of this layer is Figure 2 The feature vectors u1, v1, u2, and v2 in the text are used as examples. Taking u1 and v1 as examples, the first semantic feature vectors u1 and u2 with the [CLS] identifier are calculated using formulas (1) and (2), and the other semantic vectors v1 and v2 are formed by concatenating the features of each word except for the [CLS] identifier. v1 and v2 are a two-dimensional array. The formula is as follows:

[0094] u1=last_hidden_state[:,0] (1)

[0095] v1=last_hidden_state[:,1:-1] (2)

[0096] Other semantic vectors v1 and v2 are output to the feature interaction layer for implicit semantic extraction, and the first semantic feature vectors u1 and u2 are output to the fusion layer.

[0097] (4) Feature Interaction Layer: This layer is an interactive ABCNN model, which uses a complex attention mechanism to accurately locate the semantic focus of the text, thereby achieving accurate modeling of the semantic weight of the text context. It can capture the implicit semantic features of the text that are lost by the representational model of the feature extraction layer.

[0098] (5) Fusion Layer: This layer is based on the formula: The vector concatenation method shown concatenates semantic features u1 and u2 input to the feature extraction layer, and semantic features o1 and o2 input to the feature interaction layer, at the feature dimension level, thereby achieving semantic feature fusion. This method is compatible with feature vectors of different dimensions, even... The method can still be used for splicing of feature vectors.

[0099] Further, the initial text similarity judgment model is iteratively trained based on a transfer learning method to obtain a target text similarity judgment model, and the method specifically includes the following steps:

[0100] A1. The source domain model is iteratively trained based on the artificial annotation training set of the "yes / no type", and the trained model is stored in the local device in the form of an ordered dictionary state_dict.

[0101] A2. The network model parameters of the source domain model and the target domain model are traversed to find and record the common parameters of the two models; if the source domain model and the target domain model have common parameters, the same parameters in the target domain model are initialized using the parameter values stored in the local device of the source domain model.

[0102] In the embodiment, the pre-training Bert model in the SBert model is replaced by the Mengzi model, and the Bert model is the source domain model. The target domain model refers to the target domain in the transfer learning, and in the embodiment, the Mengzi model is the target domain model.

[0103] The transfer learning method provided in the embodiment iteratively trains the initial text similarity judgment model, and the main implementation steps of the method include: (1) the source domain model is iteratively trained based on the artificial annotation training set of the "yes / no type", and the trained model is stored in the local device in the form of an ordered dictionary state_dict; (2) the network model parameters of the source domain model and the target domain model are traversed to find and record the common parameters of the two models; (3) if the source domain model and the target domain model have common parameters, the same parameters in the target domain model are initialized using the parameter values stored in the local device of the source domain model; (4) after the model parameter initialization is completed, the loss function is minimized to realize the iterative training of the target domain model.

[0104] In the embodiment, the specific implementation method of the transfer learning realizes an English-to-Chinese transfer training. The original Bert model is mainly an English model, the Mengzi model is a modified version of the Bert model for Chinese, and the two models have common structures. When starting training, the parameters of the Mengzi model are initialized to the parameters of the Bert model in the places where the structures of the two models are consistent, that is, in the places where there are common parameters, so that the model learns the semantic similarity judgment of Chinese by referring to the pattern of English, thereby accelerating the iterative training.

[0105] A3, after the target domain model parameter initialization is completed, the Mengzi pre-training model is trained and optimized by using a CoSENT loss function; wherein the CoSENT loss function is represented as:

[0106]

[0107] The text similarity judgment always satisfies the constraint condition during the iterative training process, that is, the similarity between positive samples u i and u j is always greater than the similarity between negative samples u k and u l ; the constraint condition is shown in formula (4):

[0108] cos(u i ,u j )>cos(u k ,u l ) (4)

[0109] A4, the data enhancement method based on text editing is used to quickly multiply the rare samples in the training data set, and the data enhancement method includes any one or more of synonym random insertion, synonym random replacement, arbitrary word random exchange and arbitrary word random deletion.

[0110] To further improve the performance of the text similarity judgment model, the CoSENT loss function shown in formula (3) is used to train and optimize the Mengzi pre-training model, and the constraint condition in the training and optimization process is shown in formula (4).

[0111] To further improve the robustness of the target text similarity judgment model, the data enhancement method based on text editing is used to quickly multiply the rare samples in the unbalanced data set. The specific text editing operation includes any one or more of the following methods: (1) synonym random insertion; (2) synonym random replacement; (3) arbitrary word random exchange; (4) arbitrary word random deletion. The above synonyms come from a local synonym dictionary, and an exemplary dictionary is composed of the synonym dictionary of Harbin Institute of Technology and the high-frequency keywords in the unbalanced data set extracted by TF-IDF.

[0112] Optionally, the target text similarity calculation model is obtained by parallel heterogeneous integration of a plurality of different interactive pre-training models, including:

[0113] Roberta-large, Roberta-pair, ERNIE and Bert-wwm-ext pre-training models are parallel heterogeneous integrated into a double-layer structure model by using a stacking integration method to obtain an initial text similarity calculation model;

[0114] The initial text similarity calculation model is iteratively trained based on an adversarial training and an FP16 mixed precision method to obtain a target text similarity calculation model.

[0115] As shown in Figure 3 The initial text similarity calculation model is a layered heterogeneous ensemble model with a two-layer structure, which is formed by Roberta-large, Roberta-pair, ERNIE and Bert-wwm-ext pre-training models.

[0116] The text similarity calculation model and the primary learner with the optimal performance are in a "gradual equivalence" relationship, so the initial text similarity calculation model has a high accuracy.

[0117] After the initial text similarity calculation model is formed, the FGM adversarial training and the FP16 mixed precision method are used to iteratively train the initial text similarity calculation model to obtain a target text similarity calculation model. Since the memory of FP16 half-precision floating-point representation is only half of that of FP32 single-precision floating-point representation, using FP16 instead of FP32 to realize parameter storage and multiplication operation can accelerate the model training efficiency.

[0118] Optionally, the Roberta-large, Roberta-pair, ERNIE and Bert-wwm-ext pre-training models are formed into a double-layer structure model by using a stacked ensemble method to form an initial text similarity calculation model, comprising:

[0119] The original training set is divided into three equal parts and provided to the three primary learners Roberta-large, Roberta-pair and ERNIE pre-training models for training;

[0120] The three primary learners calculate three first text similarity values respectively by using a 5-fold cross-validation method, and the three first text similarity values are horizontally spliced to generate a secondary training set;

[0121] The secondary learner Bert-wwm-ext pre-training model divides the secondary training set into five equal parts by using a 5-fold cross-validation method, one of which is used as a validation set and the rest are used as a training set, which are provided to the Bert-wwm-ext pre-training model for training, and the model with the optimal performance is saved locally;

[0122] The BERT-wwm-ext model stored is used to calculate the text similarity on the secondary training set to obtain a second text similarity value, the first text similarity value and the second text similarity value are weighted and averaged to obtain a target text similarity value.

[0123] The three primary learners provide the secondary training set and the secondary test set for the secondary learner, and provide a calculation factor for the text similarity calculation of the second layer architecture. The specific implementation of the three primary learners to generate the secondary training set is: first, the original training set is divided into three equal parts, and each part is provided to a primary learner; second, the three primary learners use the 5-fold cross-validation method to calculate the text similarity respectively, and the text similarities are horizontally spliced to finally generate the secondary training set. The 5-fold cross-validation method is used to effectively avoid overfitting. The generation steps of the secondary test set are basically the same as the generation steps of the secondary training set described above, and the difference between the two is that the secondary test set is generated without using the 5-fold cross-validation method, but using the complete test set of each primary learner.

[0124] The secondary learner BERT-wwm-ext of the second layer still uses the 5-fold cross-validation method to divide the secondary training set into five equal parts, and uses one part as a validation set in turn, and the remaining parts as training sets. Each training round uses the divided training set to implement model training, and saves the model with the best performance locally. The BERT-wwm-ext model stored locally is used to calculate the text similarity on the secondary test set. The similarity calculation result pred second of the BERT-wwm-ext is obtained. frist The weighted sum and the mean value are calculated to obtain the final text similarity quantification calculation result, as shown in the formula:

[0125] res final = mean(w1×pred first +w2×pred second )

[0126] Wherein, res final represents the final output of the text similarity calculation model, and its value range is between 0 and 1; w1 and w2 represent the weight. In this embodiment, w1 is set to 0.8, and w2 is set to 0.2.

[0127] Optionally, the initial text similarity calculation model is iteratively trained based on the adversarial training and the FP16 mixed precision method to obtain a target text similarity calculation model, comprising:

[0128] The FGM adversarial training and the FP16 mixed precision method are used to implement the iterative training of the text similarity calculation model; the adversarial training method is shown in formula (5):

[0129]

[0130] The feature vectors of each learner before normalization are used to calculate the text similarity using the inverse function logit of sigmoid, and the calculated text similarity is respectively arithmetically averaged, geometrically averaged and harmonically averaged; wherein the inverse function logit is as shown in formula (6):

[0131]

[0132] Adversarial training can occupy a large amount of memory, prolong the model training time, and reduce the model training efficiency; the FP16 mixed precision method reduces the memory occupation and accelerates the model training efficiency.

[0133] The adversarial training and the FP16 mixed precision method add perturbation to the input to increase the loss, and minimize the loss on the parameters to improve the robustness and accuracy of the text similarity calculation model, as shown in (5).

[0134] After iterative training, in order to further improve the accuracy of the target text similarity calculation model, the inverse function logit of sigmoid shown in formula (6) is used to calculate the text similarity, and the specific operation is: the feature vectors of each learner before normalization are used to calculate the text similarity using the logit function, and the calculated text similarity is respectively arithmetically averaged, geometrically averaged and harmonically averaged. Among them, the arithmetic average is used to describe the central tendency of the text semantic features, the geometric average is used to eliminate the negative effect of extreme value on the arithmetic average result, and the harmonic average is equivalent to adding a penalty mechanism.

[0135] Optionally, on the basis of the above, further comprising:

[0136] The positive and negative samples are sampled in proportion to supplement the secondary training set;

[0137] The proportion of sampling positive samples is represented as:

[0138] num pos =floor(len(temp falses )×rate origin ) (7)

[0139] The proportion of sampling negative samples is represented as:

[0140] num neg =floor(len(temp trues )÷rate origin ) (8)

[0141] wherein, num pos is the sampling number of positive samples, num neg is the sampling number of negative samples, temp falses and temp trues are lists respectively storing newly added positive and negative samples, rate origin is the original ratio of the number of positive and negative samples.

[0142] To further improve the robustness of the text similarity calculation model, data augmentation is realized based on the transitive property of text similarity. To ensure the rationality of the expansion operation, formula (7) and formula (8) are used to realize proportional sampling of newly added positive and negative samples, and finally realize fast and high-quality expansion of small sample datasets and unbalanced datasets.

[0143] The technical scheme of the embodiment of the application constructs a target text similarity judgment model and a target text similarity calculation model based on residual fusion and stacked integration method, wherein the target text similarity judgment model is obtained by fusing the SBERT model of the representation type and the ABCNN model of the interaction type, and the target text similarity calculation model is obtained by parallel heterogeneous integration of multiple different interaction type pre-training models. It is suitable for text similarity judgment and text similarity calculation, solves the technical problem that the current text similarity analysis model is difficult to have high retrieval efficiency and accuracy, and realizes text similarity judgment with accuracy and retrieval efficiency, and high-accuracy text similarity calculation.

[0144] Embodiment two

[0145] A text similarity analysis system based on residual fusion and stacked integration, comprising:

[0146] A text similarity judgment module is configured to input the text to be analyzed into the target text similarity judgment model for similarity judgment, and obtain a judgment result; the target text similarity judgment model is obtained by fusing the SBERT model of the representation type and the ABCNN model of the interaction type.

[0147] A text similarity calculation module is configured to input the text to be analyzed into the target text similarity calculation model for similarity calculation if the text to be analyzed is judged as similar text, and obtain a target text similarity value; the target text similarity calculation model is obtained by parallel heterogeneous integration of multiple different interaction type pre-training models.

[0148] The text similarity analysis system based on residual fusion and stacked integration provided by the embodiment of the application can execute the text similarity analysis method based on residual fusion and stacked integration provided by any embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method.

[0149] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A text similarity analysis method based on residual fusion and stacked ensemble, characterized in that, The application relates to a text similarity judgment method and device. If the text to be analyzed is judged as similar text, the text to be analyzed is input into a target text similarity calculation model to perform similarity calculation, and a target text similarity value is obtained; the target text similarity calculation model is obtained by parallel heterogeneous integration of multiple different interactive pre-training models. The method comprises the following steps: The text similarity judgment model is constructed and comprises an input layer, an embedding layer, a feature extraction layer, a feature interaction layer, a fusion layer and an output layer. The input layer pre-processes the training text and manually labels the training text based on a yes-no type; The embedding layer performs text vectorization on the labeled training text and generates vector features of the training text through a Mengzi tokenizer; The feature extraction layer adopts an SBERT model, and a BERT pre-training model in the SBERT model is replaced by a lightweight Mengzi pre-training model; the vector features are provided as standard input to the Mengzi pre-training model to obtain a fixed-dimension dense feature vector; The output layer comprises sequentially connected full connection layers, a normalization layer, an activation function, a full connection layer and a softmax function, and a text similarity judgment result is obtained according to a vector after semantic feature fusion. The feature interaction layer adopts an interactive ABCNN model, and the feature interaction layer fully interacts the features extracted by the SBERT model according to an attention mechanism to extract deep second semantic feature vectors and ; The fusion layer fuses the first semantic feature vector input by the feature extraction layer based on a vector splicing method and and the second semantic feature vector input by the feature interaction layer and splices in the feature dimension level to obtain a fused semantic feature vector The vector features are provided as standard input to the Mengzi pre-training model to obtain a fixed-dimension dense feature vector, which comprises:

2. The text similarity analysis method of claim 1, wherein, The initial text similarity judgment model is iteratively trained based on a transfer learning method to obtain a target text similarity judgment model. After the vector features are provided as standard input to the Mengzi pre-training model, the first semantic feature vector with [CLS] identifier is calculated through formulas (1) and (2) and , and other semantic vectors formed by splicing the features of each word and ; the formula is as follows: (1) (2) Other semantic vectors and The output is sent to the feature interaction layer for implicit semantic extraction, and the first semantic feature vector is... and Output to the fusion layer.

3. The text similarity analysis method of claim 1, wherein, Iterative training of a source domain model is realized based on a yes-no type artificial labeling training set, and the trained model is stored in the local area in the form of an ordered dictionary state_dict; Network model parameters of the source domain model and the target domain model are traversed to find and record common parameters; if the source domain model and the target domain model have common parameters, the parameter values stored in the local area of the source domain model are used to initialize the same parameters in the target domain model. After the target domain model parameter initialization is completed, the Mengzi pre-training model is trained and optimized by using a CoSENT loss function; wherein the CoSENT loss function is expressed as: A data enhancement method based on text editing is used to rapidly multiply the rare samples in the training data set, and the data enhancement method comprises any one or more of synonym random insertion, synonym random replacement, arbitrary word random exchange and arbitrary word random deletion. （3） The text similarity judgment always meets a constraint condition in the iterative training process, that is, the similarity between positive samples and is always greater than the similarity between negative samples and ; the constraint condition is shown in formula (4): （4） ​ 4. The text similarity analysis method of claim 1, wherein, The target text similarity calculation model is obtained by parallel heterogeneous integration of multiple different interactive pre-training models, including: The Roberta-large, Roberta-pair, ERNIE and Bert-wwm-ext pre-training models are parallel heterogeneous integrated into a double-layer structure model by using a stacking ensemble method to obtain an initial text similarity calculation model; The initial text similarity calculation model is iteratively trained based on an adversarial training and FP16 mixed precision method to obtain a target text similarity calculation model.

5. The text similarity analysis method of claim 4, wherein, The Roberta-large, Roberta-pair, ERNIE and Bert-wwm-ext pre-training models are parallel heterogeneous integrated into a double-layer structure model by using a stacking ensemble method to obtain an initial text similarity calculation model, including: The original training set is divided into three equal parts and provided to three primary learners, Roberta-large, Roberta-pair and ERNIE pre-training models for training; The three primary learners use a 5-fold cross-validation method to calculate three first text similarity values, and horizontally splice them to generate a secondary training set; A secondary learner, the Bert-wwm-ext pre-training model, divides the secondary training set into five equal parts using a 5-fold cross-validation method, one of which is used as a validation set and the rest as a training set, which is provided to the Bert-wwm-ext pre-training model for training. The model with the best performance is saved locally; The stored BERT-wwm-ext model is used to calculate the text similarity on the secondary training set to obtain a second text similarity value. The first and second text similarity values are weighted and summed to obtain a target text similarity value.

6. The text similarity analysis method of claim 4, wherein, The initial text similarity calculation model is iteratively trained based on an adversarial training and FP16 mixed precision method to obtain a target text similarity calculation model, including: FGM adversarial training and FP16 mixed precision method are used to realize the iterative training of the text similarity calculation model; the adversarial training method is shown in formula (5): （5） The feature vectors of each learner before normalization are used for text similarity calculation using the inverse function logit of sigmoid, and the calculated text similarity is arithmetically averaged, geometrically averaged and harmonically averaged; wherein the inverse function logit is shown in formula (6): （6）。 7. The text similarity analysis method of claim 5, wherein, Also including: The secondary training set is supplemented by proportionally sampling positive and negative samples; The proportionally sampled positive samples are represented as: （7） The proportionally sampled negative samples are represented as: （8） wherein, is the number of samples of positive samples, is the number of samples of negative samples, and are lists storing the newly added positive and negative samples, respectively, is the original ratio of the number of positive and negative samples.

8. A text similarity analysis system based on residual fusion and stacked ensemble, characterized in that, Including: A text similarity judgment module is used to input the text to be analyzed into the target text similarity judgment model for similarity judgment to obtain a judgment result. An identity mapping method of a residual network is used to fuse an SBERT model representing type and an ABCNN model representing interaction to obtain an initial text similarity judgment model. The initial text similarity judgment model is iteratively trained based on a transfer learning method to obtain a target text similarity judgment model. The text similarity calculation module is configured to, if the text to be analyzed is determined as similar text, input the text to be analyzed into a target text similarity calculation model to perform similarity calculation, and obtain a target text similarity value; the target text similarity calculation model is obtained by parallel heterogeneous integration of multiple different interactive pre-training models; The text similarity judgment model is obtained by fusing the representation type SBERT model and the interactive ABCNN model using the identity mapping method of the residual network, and includes: The text similarity judgment model is constructed and includes an input layer, an embedding layer, a feature extraction layer, a feature interaction layer, a fusion layer, and an output layer. The input layer pre-processes the training text and manually labels the training text based on a non-type; The embedding layer vectorizes the labeled training text and generates vector features of the training text through a Mengzi tokenizer; The feature extraction layer uses the representation type SBERT model, and replaces the BERT pre-training model in the SBERT model with a lightweight Mengzi pre-training model; the vector features are provided as standard input to the Mengzi pre-training model to obtain a fixed-dimension dense feature vector; The feature interaction layer adopts an interactive ABCNN model, and the feature interaction layer fully interacts the features extracted by the SBERT model according to an attention mechanism to extract deep second semantic feature vectors and ; The fusion layer fuses the first semantic feature vector input by the feature extraction layer based on a vector splicing method and The second semantic feature vector input by the feature interaction layer and The semantic feature vectors are spliced at the feature dimension level to obtain a fused semantic feature vector. The output layer includes sequentially connected fully connected layers, a normalization layer, an activation function, a fully connected layer, and a softmax function, and obtains a text similarity judgment result according to the vector after semantic feature fusion.

Images