Difference text screening method, device, equipment and storage medium

Abstract

The present application discloses a method, device, equipment and storage medium for screening differential texts. The present application pre-configures a similarity matching model for calculating the similarity of every two texts in a text collection to be screened. During model training, if it is determined that the length difference between two training texts in a training text pair exceeds a set difference threshold, the similarity label of the training text pair is defined as: the similarity with the largest median similarity between each candidate text after segmenting the longer training text and the shorter training text. Accordingly, the impact of text length differences on similarity labels can be reduced, and the similarity matching model obtained through training is more accurate, and the similarity between texts determined based on the similarity matching model will also be more accurate. Based on the similarity between texts, a target text collection whose differences meet the set difference conditions is screened out from the text collection as the final high-quality text collection.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and more specifically, to a difference text screening method, device, equipment and storage medium. Background Art

[0002] With the development of machine learning, neural network models based on machine learning technology are being applied more and more widely in various fields. Neural network models can handle a variety of tasks, such as classification tasks, labeling tasks, generation tasks, etc. In the process of training neural network models, a large amount of labeled data is generally required. If there are a large number of repeated and similar labeled texts in the labeled data, on the one hand, it will lead to repeated labeling and waste of human resources. On the other hand, the model will repeatedly learn on similar data, which will cause the model to overfit the current data in the training and testing phase, and lack generalization ability in actual use.

[0003] In order to most effectively reduce the annotation cost and improve the generalization ability of the model, this application believes that if the unlabeled data is screened to form a small amount of data sets with large text differences, and then manually annotated, a high-quality data set can be constructed. This can not only reduce repeated annotation work and shorten the annotation time, but also give the model a stronger generalization ability to the greatest extent with the same amount of data. In addition, even for the labeled data, if it can be screened to form a certain amount of data sets with large text differences, as a high-quality data set, it can give the model a stronger generalization ability to the greatest extent.

[0004] Therefore, it is necessary to develop a differential text screening solution that can quickly select a small amount of high-quality data from the text to be annotated or the annotated text for subsequent processing. Summary of the invention

[0005] In view of the above problems, this application is proposed to provide a method, device, equipment and storage medium for differential text screening to achieve the purpose of screening high-quality data sets for training neural network models. The specific scheme is as follows:

[0006] In a first aspect, a method for screening difference texts is provided, comprising:

[0007] Get the text collection to be filtered;

[0008] Calculating the similarity between every two texts in the text collection using a preconfigured similarity matching model;

[0009] The similarity matching model is trained using training text pairs annotated with similarity labels as training data, wherein if the length difference between two training texts in a training text pair exceeds a set difference threshold, the similarity label of the training text pair is: the similarity with the largest median similarity between each candidate text after segmenting the longer training text and the shorter training text;

[0010] Based on the similarity between any two texts in the text set, a target text set is screened out from the text set, and the differences between the texts in the target text set meet a set difference condition.

[0011] In a second aspect, a difference text screening device is provided, comprising:

[0012] A text collection acquisition unit, used to acquire a text collection to be screened;

[0013] A similarity calculation unit, used for calculating the similarity between every two texts in the text set by using a preconfigured similarity matching model;

[0014] The similarity matching model is trained using training text pairs annotated with similarity labels as training data, wherein if the length difference between two training texts in a training text pair exceeds a set difference threshold, the similarity label of the training text pair is: the similarity with the largest median similarity between each candidate text after segmenting the longer training text and the shorter training text;

[0015] The text set screening unit is used to screen out a target text set from the text set based on the similarity between two texts in the text set, wherein the differences between the texts in the target text set meet the set difference conditions.

[0016] In a third aspect, a differential text screening device is provided, comprising: a memory and a processor;

[0017] The memory is used to store programs;

[0018] The processor is used to execute the program to implement the various steps of the difference text screening method described above.

[0019] In a fourth aspect, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, each step of the above difference text screening method is implemented.

[0020] By means of the above technical scheme, the present application pre-configures a similarity matching model for calculating the similarity of each two texts in the text set to be screened, and the similarity matching model is trained with training text pairs annotated with similarity labels as training data. Considering that some information may be covered between texts, for example, the information expressed by one text is completely covered in another text, but due to the large difference in the length of the two texts, there may be a low similarity between the two texts calculated. If the similarity matching model is trained based on such training data, it will lead to the problem of poor model effect. For this reason, the present application defines the similarity label of the training text pair as: the similarity of the candidate texts after the longer training text is segmented, and the similarity median value of the shorter training text is the largest. Accordingly, the influence of the text length difference on the similarity label can be reduced, so that the similarity label in the training data is more accurate, and the similarity matching model obtained by training is more accurate, and the similarity between the two texts in the text set determined based on the similarity matching model will also be more accurate. Furthermore, based on the similarity between two texts, a target text set whose difference meets the set difference condition is screened out from the text set as the final high-quality text set.

[0021] It can be seen that if the text set to be screened contains unlabeled data, the high-quality text set obtained by screening can be marked by the user, and after marking, it can be used as training data for the neural network model in the downstream task; if the text set to be screened contains labeled data, the high-quality text set obtained by screening can be directly used as training data for the neural network model in the downstream task. According to the scheme of the present application, it is possible to reduce repeated labeling work, shorten the labeling time, and give the model stronger generalization ability to the greatest extent under the same amount of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 A schematic diagram of a process flow of a method for screening difference texts provided in an embodiment of the present application;

[0024] Figure 2a This is a schematic diagram showing the projection of a sample set into two-dimensional space.

[0025] Figure 2bAn example of a schematic diagram of the projected text after filtering into a two-dimensional space;

[0026] Figure 3 A structural diagram of a similarity matching model is illustrated;

[0027] Figure 4 A structural diagram of another similarity matching model is illustrated;

[0028] Figure 5 A schematic diagram of selecting a target subvector through a sliding window is shown as an example;

[0029] Figure 6 A flowchart illustrating a differential text screening solution is provided;

[0030] Figure 7 A schematic diagram of the structure of a difference text screening device disclosed in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of the structure of a differential text screening device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The present application provides a difference text screening scheme, which can screen a target text set from a text set to be screened, and the differences between the texts in the target text set meet the set difference conditions. The present application scheme can be applied to the determination process of the training data of the neural network model. For example, a large amount of unlabeled data can be used as a text set to be screened, and the target text set obtained after screening according to the present application scheme can be marked by the user, which can not only reduce repeated marking work and shorten the marking time, but also give the model a stronger generalization ability to the greatest extent under the same amount of data. For another example, the data that has been labeled can be used as a text set to be screened. Since there may be a large number of repeated or similar texts in the labeled data in the text set, such data will cause the neural network model to overfit the current data in the training and testing phase, and lack generalization ability in actual use. For this reason, the target text set is obtained after screening the text set to be screened according to the present application scheme, and the text differences in the target text set are stronger. The neural network model trained based on this can give the model a stronger generalization ability.

[0034] Of course, the above examples only illustrate several optional application scenarios of the difference text screening solution. In addition, the present application solution can also be applied to other scenarios that require screening of difference texts.

[0035] The present application solution can be implemented based on a terminal with data processing capabilities, which can be a mobile phone, computer, server, cloud, etc.

[0036] Next, combine Figure 1 The difference text screening method of the present application may include the following steps:

[0037] Step S100: Obtain a text set to be screened.

[0038] Specifically, for a scenario where differential text screening is required, the text set that needs to be screened in the scenario may be used as the text set to be screened.

[0039] For example, in the medical field, facing a large amount of case data, it is necessary to annotate the large amount of case data before training the medical task model. In order to reduce the cost of manual annotation and improve the quality of model training data, the large amount of case data can be used as a text set to be screened.

[0040] Step S110: Calculate the similarity between every two texts in the text set using a preconfigured similarity matching model.

[0041] The similarity matching model is trained using training text pairs annotated with similarity labels as training data. The similarity labels can be in the form of similarity values ​​or similarity levels. The similarity between two texts can be calculated using a similarity calculation method.

[0042] Considering that some texts may contain information, for example, the information expressed by one text is completely contained in another text, but due to the large difference in the length of the two texts, the calculated similarity between the two texts may be low. If the similarity matching model is trained based on such training data, the model effect will be poor. For this reason, in this embodiment, for the training data of the similarity matching model, if the length difference between the two training texts in a training text pair exceeds the set difference threshold, the similarity label of the training text pair can be defined as:

[0043] The similarity between each candidate text after segmenting the longer training text in the training text pair and the shorter training text in the training text pair having the largest median similarity.

[0044] Of course, if the length difference between the two training texts in the training text pair does not exceed the set difference threshold, the similarity label of the training text pair can be directly represented by the similarity between the two training texts.

[0045] According to the definition method of the similarity labels of the text pairs introduced in this embodiment, for the training text pairs with large differences in text length, their similarity can be accurately measured, so that the definition of the similarity labels in the training data is more accurate, and the similarity matching model based on this training is more effective.

[0046] Step S120: Filter out a target text set from the text set based on the similarity between any two texts in the text set.

[0047] The differences between the texts in the target text set meet the set difference conditions.

[0048] It is understandable that the purpose of the embodiment of the present application is to filter out texts with greater differences. Therefore, by setting the difference condition, the differences between the texts in the target text set that are finally filtered out are greater.

[0049] In the embodiment of the present application, a similarity matching model is pre-configured for calculating the similarity of each two texts in the text set to be screened, and the similarity matching model is trained with training text pairs annotated with similarity labels as training data. Considering that some information may be covered between texts, for example, the information expressed by one text is completely covered in another text, but due to the large difference in the length of the two texts, the similarity between the two texts may be calculated to be low. If the similarity matching model is trained based on such training data, the model effect will be poor. For this reason, the present application defines the similarity label of the training text pair as: the similarity of the candidate texts after the longer training text is segmented and the median similarity of the shorter training text is the largest. Accordingly, the influence of the text length difference on the similarity label can be reduced, so that the similarity label in the training data is more accurate, and the similarity matching model obtained by training is more accurate, and the similarity between the two texts in the text set determined based on the similarity matching model will also be more accurate. Furthermore, based on the similarity between two texts, a target text set whose difference meets the set difference condition is screened out from the text set as the final high-quality text set.

[0050] It can be seen that if the text set to be screened contains unlabeled data, the high-quality text set obtained by screening can be marked by the user, and after marking, it can be used as training data for the neural network model in the downstream task; if the text set to be screened contains labeled data, the high-quality text set obtained by screening can be directly used as training data for the neural network model in the downstream task. According to the scheme of the present application, it is possible to reduce repeated labeling work, shorten the labeling time, and give the model stronger generalization ability to the greatest extent under the same amount of data.

[0051] In some embodiments of the present application, the training process of the similarity matching model is described.

[0052] First, the process of obtaining training data for the similarity matching model is introduced, which includes the following steps:

[0053] S1. Selecting a number of text pairs from a sample set, wherein the texts contained in the sample set and the text set to be screened belong to the same field.

[0054] It is understandable that in order to ensure that the trained similarity matching model can perform text screening on the text set to be screened, when training the similarity matching model, the sample set serving as the source of training data should contain texts belonging to the same field as the text set to be screened.

[0055] In an optional case, the sample set may be a subset of the text set to be screened.

[0056] For example, if the text set to be screened is a massive amount of electronic medical record data in a medical scenario, a portion of the massive amount of electronic medical record data can be selected as a sample set.

[0057] The sample set contains multiple texts. In this step, several text pairs are selected from the sample set.

[0058] S2. Calculate the similarity label of each text pair:

[0059] If the length difference between the two texts in the text pair does not exceed the set difference threshold, the similarity between the two texts is calculated according to the set similarity calculation method as the similarity label of the text pair.

[0060] If the length difference between the two texts in the text pair exceeds the set difference threshold, the longer text of the two texts is segmented to obtain several candidate texts, and the similarity between each candidate text and the shorter text of the two texts is calculated according to the set similarity calculation method, and the similarity with the largest median value between each candidate text and the shorter text is selected as the similarity label of the text pair.

[0061] There are many ways to set the similarity calculation method, for example:

[0062] Similarity calculation methods based on text embedding representation, similarity calculation methods based on TF-IDF (Term Frequency-Inverse Document Frequency), similarity calculation methods based on edit distance, etc. are introduced in detail below.

[0063] It should be noted that different similarity calculation methods measure different indicators. In some cases, there may be two texts with overlapping information, but the calculated similarity value is very small due to the large difference in text length. The following is an example:

[0064] Text 1: After having a full dinner yesterday, the patient felt chest tightness and intermittent colic in the upper abdomen, followed by nausea and vomiting. He then suddenly vomited about 600ml of coffee-like liquid and had a black stool of about 400ml. At that time, the patient was pale and sweating profusely, with general weakness and dizziness, but no fever, syncope, blackness, vomiting of blood, bloody stools, etc. His family members did not pay much attention to him.

[0065] Text 2: The patient suffers from chest tightness, intermittent colic in the upper abdomen, nausea and vomiting.

[0066] Analysis of the two texts shows that the information expressed in text 1 can cover the information expressed in text 2. However, due to the difference in the length of the two texts, the calculated similarity may be very low. If we assume that the two texts are very different and annotate them at the same time, it will inevitably increase the amount of annotation and lead to repeated annotation.

[0067] In order to reduce the impact of text length differences on similarity calculation, in this embodiment, the text length differences are divided, and different methods are used to calculate the similarity labels of text pairs when the text length differences are different.

[0068] The text length difference can be represented by the length difference of the two texts. In addition, it can also be represented by the ratio of the length difference of the two texts to the length sum of the two texts. Taking the latter as an example, the corresponding difference threshold can be 1 / 6 or other set parameters.

[0069] This application may define that when the text length difference does not exceed a set difference threshold, it indicates that the text lengths are similar; otherwise, it indicates that the text length difference is large.

[0070] If the length difference between the two texts in the text pair does not exceed the set difference threshold, the similarity between the two texts can be calculated according to the set similarity calculation method, and the result is used as the similarity label of the text pair.

[0071] If the length difference between the two texts in the text pair exceeds the set difference threshold, it means that the length difference is large. For the convenience of expression, the longer text in the text pair is defined as the longer text, and the other text is defined as the shorter text. At this time, the longer text can be segmented to obtain several candidate texts, and the similarity between each candidate text and the shorter text is calculated according to the set similarity calculation method, and the similarity with the largest median value of the similarities between each candidate text and the shorter text is selected as the similarity label of the text pair.

[0072] When segmenting a longer text, the segmentation can be performed at the punctuation position in the longer text. If there is no punctuation in the longer text, or the length of the segmented text is still significantly different from the length of the shorter text, the character string in the longer text can be directly segmented according to the set length. The set length can be the length of the shorter text, or the length of the shorter text ± Θ, where Θ represents a set parameter.

[0073] It is understandable that when a longer text is segmented according to a set length, the length of the last candidate text after segmentation may be too short. For example, the length difference between the last candidate text and the shorter text exceeds the set difference threshold. If this happens, the segmentation position for the last candidate text can be shifted forward until the length difference between the last candidate text and the shorter text does not exceed the set difference threshold.

[0074] S3. Using each of the text pairs and the corresponding similarity labels as training data for the similarity matching model.

[0075] Specifically, the similarity label of the text pair obtained in the previous step can be a similarity value or a similarity level. The larger the similarity value, the higher the similarity between the two texts and the smaller the difference. When the similarity label is a similarity level, it can be divided into several levels, for example, it can be divided into six levels: 0-5, where the larger the value, the higher the similarity, 0 represents irrelevant, and 5 represents very relevant.

[0076] Each text pair carrying a similarity label is used as training data for the similarity matching model.

[0077] When calculating the similarity tags of the text pairs in the above step S2, the similarity calculation method is mentioned. This embodiment introduces a similarity calculation method that integrates three indicators. For the two texts to be calculated:

[0078] S21, determining the embedding representation features of the two texts respectively, and calculating the first similarity sim1 between the two embedding representation features;

[0079] S22, determining the term frequency-inverse document frequency TF-IDF features of the two texts, and calculating the second similarity sim2 between the two TF-IDF features;

[0080] S23, calculating a third similarity sim3 based on the edit distance between the two texts;

[0081] S24, performing weighted addition on the first similarity sim1, the second similarity sim2 and the third similarity sim3, and taking the result as the final similarity score(i,j) between the two texts:

[0082] score(i,j)=α*sim1(i,j)+β*sim2(i,j)+γ*sim3(i,j)

[0083] i, j represent two different texts. α is the weight of the first similarity, β is the weight of the second similarity, and γ is the weight of the third similarity.

[0084] Among them, the first similarity sim1, the second similarity sim2 and the third similarity sim3 measure the text similarity from the semantic, word and character levels respectively.

[0085] The first similarity sim1 reflects the semantic similarity of the two texts. The second similarity sim2 reflects the similarity between the words that make up the two texts. The third similarity sim3 reflects the similarity between the two texts at the string level.

[0086] In some scenarios, the following problems may exist: 1. The semantic similarity of different texts in the same scenario is not very different, such as texts in medical scenarios. At this time, if only sim1 is used as the final similarity, the similarity calculation result will be inaccurate; 2. For texts with the same semantics, due to the diversity of expressions, the similarity at the word and character level is very low. At this time, if only sim2 or sim3 is used as the final similarity, the similarity calculation result will also be inaccurate. For this reason, in this embodiment, three indicators are measured simultaneously to calculate the similarity of two texts, so that the final similarity calculation result is more accurate.

[0087] In order to more reasonably select α, β, and γ, an optional method for determining the three weights is introduced in the embodiment of the present application.

[0088] It should be noted that N texts with relatively large similarity differences can be selected from the sample set according to the similarity. The present application can project the sample set and the selected N texts into a two-dimensional space {x i ,y i},i∈[i,N]. It is understandable that the distribution of the N texts selected in the sample set should be as dispersed as possible and relatively balanced. To this end, in this embodiment, a dynamic assignment method can be used to assign values ​​to α, β, and γ, and the distribution density of the N texts in the two-dimensional space is used as an evaluation index. The weight combination of α, β, and γ that makes the distribution density most balanced is selected as the final weight result.

[0089] Examples include Figure 2a and Figure 2b As shown:

[0090] Figure 2a The example is a schematic diagram of the sample set projected into two-dimensional space. Figure 2b This is a schematic diagram of the N texts finally selected according to a certain weight combination method after being projected into the two-dimensional space. It can be seen that under this weight combination method, the distribution of the N selected texts is relatively dispersed and balanced.

[0091] Next, the weight determination process is introduced in detail:

[0092] S1. Using a plurality of different weight combinations as weights of the first similarity, the second similarity and the third similarity, and calculating the similarities of any two text pairs in the sample set according to a set similarity calculation method under each weight combination.

[0093] There are many ways to combine the weights of α, β, and γ. For example, under the premise that the sum of α, β, and γ is 1, α, β, and γ are separated by a certain value, such as 0.1, and parameters are selected between 0 and 1.

[0094] S2. Under each weight combination:

[0095] According to the calculated similarities between any two text pairs in the sample set, N texts whose differences meet the set difference conditions are selected from the sample set; the sample set and the N texts are projected into a two-dimensional space, and based on the distribution of the N texts and the sample set in the two-dimensional space, the balance of the text distribution density corresponding to the weight combination is determined.

[0096] Specifically, there may be multiple ways to select N texts whose differences satisfy the set difference conditions from the sample set. This embodiment illustrates an optional implementation method:

[0097] It is defined that there are a total of M texts in the sample set. After obtaining the similarity between two texts, there are M-1 similarities for each text. The sum of the M-1 similarities is taken as the similarity value between the text and the overall sample set.

[0098] Sort the similarity value of each text in the sample set relative to the sample set in order from small to large, and select the first N texts with smaller values.

[0099] In order to measure the distribution density of the selected N texts compared to the sample set, the sample set and the N texts are projected into a two-dimensional space in this step. During the projection, the feature representation of the text can be extracted, such as the embedding feature representation, the TF-IDF feature representation, etc., and then projected into the two-dimensional space according to the feature representation of the text. In the two-dimensional space, the balance of the text distribution density corresponding to each weight combination is determined.

[0100] An optional way to determine the balance of text distribution density corresponding to each weight combination is as follows:

[0101] S21, projecting each text in the sample set into a two-dimensional space to obtain an original sample space, and dividing the original sample space into a plurality of sub-areas.

[0102] S22, projecting the N texts into a two-dimensional space, and counting the number of texts projected into each of the sub-regions to obtain a text distribution density set.

[0103] S23, calculating the variance of the text distribution density set as the text distribution density balance corresponding to the weight combination.

[0104] Among them, the smaller the variance, the better the balance of text distribution density.

[0105] S3. Select a weight combination with the best text distribution density balance as the final weight of each of the first similarity, the second similarity and the third similarity.

[0106] Specifically, after calculation in the previous step, the text distribution density balance corresponding to each weight combination can be obtained, and then the weight combination with the best text distribution density balance can be selected as the final weights of the first similarity, the second similarity and the third similarity.

[0107] In this embodiment, a dynamic assignment method is adopted, and the distribution density of the selected N texts is used as an indicator to evaluate the quality of the weight combination. Finally, the weight combination with the best distribution density balance is selected as the final weight of each of the three indicators in the similarity calculation formula, thereby ensuring the rationality of the designed similarity calculation formula.

[0108] In some embodiments of the present application, the structure of the similarity matching model is further introduced.

[0109] like Figure 3 As shown, the similarity matching model may include a semantic representation acquisition module, a local representation acquisition module, a feature fusion module and a similarity prediction module.

[0110] The semantic representation acquisition module is used to calculate the semantic feature representation of each text for each of the two input texts.

[0111] Specifically, the two input texts can be pre-trained word vectors, and the words in the texts can be converted into vector form to obtain the respective embedded representations of the two texts, and then the respective embedded representations of the two texts are input into the semantic representation acquisition module.

[0112] The local representation acquisition module is used to calculate the local feature representation between two input texts.

[0113] Specifically, in view of the length difference between the input texts, in order to avoid the influence of the length difference on the similarity result, in this embodiment, a local representation acquisition module is added to the similarity matching model to obtain the local feature representation between the two input texts.

[0114] The feature fusion module is used to calculate the fused feature representation of the semantic feature representation of each of the two input texts and the local feature representation. That is, the feature fusion module fuses the feature representations calculated by the semantic representation acquisition module and the local representation acquisition module, so that the fused feature representation contains both the semantic information of each input text and the local feature information between the two input texts.

[0115] The similarity prediction module is used to predict the similarity between texts based on the fused feature representation calculated by the feature fusion module.

[0116] In this embodiment, when the similarity matching model performs similarity prediction, it can be regarded as a multi-classification problem, that is, the similarity matching model predicts the level of similarity between two input texts. Therefore, a cross entropy loss function can be used for training during the model training process.

[0117] The structure of the similarity matching model provided in this embodiment can simultaneously obtain the semantic features of the two input texts and the local features between the two texts, and then fuse the obtained features to predict the similarity between the input texts based on the fused feature representation. Since the local features between the texts are taken into account, the problem of inaccurate similarity prediction caused by differences in text length is avoided.

[0118] Further optionally, the semantic representation acquisition module may include a semantic representation extraction module and a first attention module. On this basis, the process of the semantic representation acquisition module calculating the semantic feature representations of the two texts may include:

[0119] The primary semantic feature representation of each of the two texts is obtained through the semantic representation extraction module, and the high-level semantic feature representation is further determined based on the primary semantic feature representation of each text through the first attention module.

[0120] Combination Figure 4 As shown:

[0121] Input text 1 can be represented as:

[0122] Input text 2 can be represented as:

[0123] in, and are the words in input text 1 and input text 2 respectively, and define K>L.

[0124] Figure 4 In the example structure, the semantic representation extraction module can adopt a bidirectional long short-term memory network (Bi-directional Long Short Term Memory, BiLSTM) structure. Of course, in addition to this, the semantic representation extraction module can also adopt other optional feature extraction network structures.

[0125] The input text is converted into vector form, and the embedding representations embedding(s1) and embedding(s2) are obtained respectively. The embedding representations of the two texts are input into BiLSTM respectively, and the hidden states of the forward and reverse LSTM at each time step are taken on the two input texts respectively, and the feature matrices X1 and X2 extracted by the bidirectional LSTM network are concatenated as the input of the first attention module.

[0126] X1=BiLSTM(embedding(s1))

[0127] X2=BiLSTM(embedding(s2))

[0128] Furthermore, the first attention module Self-Attention extracts higher-level semantic features. Each hidden state performs attention calculation with all states in the text. The purpose is to learn the word dependency relationship within the text and capture the internal structure of the sentence. Finally, the high-level semantic feature representations of the two texts are obtained respectively:

[0129] a=SelfAttention(X1)

[0130] b=SelfAttention(X2)

[0131] Further optionally, the local representation acquisition module may include a sub-vector selection module and a second attention module. On this basis, the process of calculating the local feature representation between two texts through the local representation acquisition module may include:

[0132] Through the sub-vector selection module is the window size, slide on the primary semantic feature representation corresponding to the longer of the two texts to obtain a subvector, and multiply the obtained subvector with the primary semantic feature representation corresponding to the shorter of the two texts to obtain the vector multiplication result corresponding to each subvector; select the target subvector x corresponding to the vector multiplication result of the maximum value; L is the length of the shorter text, To set the parameters, a random integer between 0 and 2 / 5*L can be chosen.

[0133] Combination Figure 5 As shown:

[0134] Define the feature matrices X1 and X2 calculated by the above semantic representation extraction module as follows:

[0135]

[0136]

[0137] In order to better integrate the local features between texts and reduce the impact of text length differences, this step can be is the window size, slide on the feature matrix X1 of the longer text 1 to obtain the subvector, and multiply the obtained subvector with the feature matrix X2 corresponding to the shorter text 2 to obtain the vector multiplication result corresponding to each subvector. Select the target subvector x corresponding to the vector multiplication result of the maximum value.

[0138]

[0139] i = argmax(sim(i))

[0140]

[0141] in, Represents the subvector obtained by sliding on the feature matrix X1 of the longer text 1.

[0142] Through the second attention module, a local feature representation c between two texts is determined based on the target sub-vector x.

[0143] c = SelfAttention(x)

[0144] Combination Figure 4 As shown:

[0145] The local representation acquisition module is Figure 4 , which corresponds to the Cross-Feature+Self-Attention module, where Cross-Feature represents the sub-vector selection module and Self-Attention represents the second attention module.

[0146] Feature fusion module corresponds to Figure 4 The Concatation module in the example combines the three features a, b, and c obtained above, and inputs the combined feature representation into the similarity prediction module for similarity prediction.

[0147] Figure 4 In the example, the similarity prediction module adopts the Softmax structure and outputs the final similarity Y between the two texts.

[0148] Furthermore, the calculation process of the feature fusion module for calculating the fused feature representation of the semantic feature representation and the local feature representation of the two texts introduced in the above embodiment is described in detail.

[0149] In order to better fuse the two semantic feature representations a and b, and the local feature representation c, in this embodiment, a combination of vector multiplication and subtraction operations can be used, where vector multiplication represents the associated features between the two features, and vector subtraction represents the difference features between the two features. The processed vectors are further concatenated. The specific calculation process may include:

[0150] Calculate the first association feature a*b and the first difference feature ab between the semantic feature representations of the two texts, calculate the second association feature c*b and the second difference feature cb between the semantic feature representation of the shorter text of the two texts and the local feature representation; concatenate the semantic feature representations of the two texts, the first association feature, the first difference feature, the second association feature and / or the second difference feature to obtain a fused feature representation.

[0151] Among them, a*b represents the association relationship between the semantic features of the two texts, and ab represents the difference relationship between the semantic features of the two texts. c is the feature extracted from the longer text in the two texts, so c*b represents the association relationship between the two text parts, and cb represents the difference relationship between the two text parts.

[0152] In an optional case, the features obtained by the above calculations are concatenated, and the final concatenated fusion feature representation example is: [a; b; ab; a*b; cb; c*b].

[0153] The fused features cover the semantic features of the two texts, the associations and differences between the two texts, and the associations and differences between the two texts. When similarity prediction is performed based on the fused features, the similarity between the two texts can be predicted more accurately, thereby improving the accuracy of similarity prediction.

[0154] In some embodiments of the present application, the aforementioned step S120 is described as to the process of screening out a target text set from the text set based on the similarity between any two texts in the text set.

[0155] This embodiment introduces several optional implementation methods, and the examples are as follows:

[0156] The first one,

[0157] It is defined that there are a total of P texts in the text set. After obtaining the similarity between two texts, there are P-1 similarities for each text. The sum of the P-1 similarities is taken as the similarity value between the text and the overall text set.

[0158] Sort the similarity value of each text in the text set relative to the text set in ascending order, and select the first Q texts with smaller values ​​to form the target text set.

[0159] The value of Q can be set by the user.

[0160] The second type

[0161] First, based on the similarity between two texts in the text collection, a fully connected graph E is constructed.

[0162] A node in the fully connected graph E corresponds to a text in the text set. Each node is connected to other nodes through an edge, and the weight of the edge is the similarity between the two texts corresponding to the two nodes connected by the edge.

[0163] The similarity may be a similarity value or a similarity level.

[0164] Furthermore, according to the set selection rules, a path with the smallest total edge weight is selected from the fully connected graph E, and the target text set is composed of the texts corresponding to the nodes on the selected path.

[0165] It is understandable that the smaller the edge weight is, the lower the similarity between the corresponding two texts is, the greater the text difference is, and the more it meets our standard for selecting high-quality texts. To this end, in this embodiment, a path with the smallest total edge weight can be selected from the fully connected graph, and the texts corresponding to each node on the path form a target text set.

[0166] Of course, in actual application scenarios, users generally set selection rules, such as the total number of texts in the target text set to be selected, the length requirement of the texts in the target text set, etc. Then, when selecting in the fully connected graph E, selection can be made according to the set selection rules.

[0167] When selecting a specific algorithm, a variety of different algorithms can be used for processing, for example, the global optimal algorithm can be used: genetic algorithm, simulated annealing algorithm, particle swarm algorithm, etc., and non-global optimal algorithms can also be used, such as: gradient descent algorithm, greedy algorithm, etc. Compared with the global optimal algorithm, the latter has a faster calculation speed.

[0168] Taking the greedy algorithm as an example, the processing logic of the greedy algorithm is that when solving a problem, it always makes the best or optimal (i.e., most favorable) choice at the moment, hoping that the final result will be the best or optimal. The process of selecting the target text set in the fully connected graph E according to the set selection rules according to the greedy algorithm may include:

[0169] S1. Construct an empty graph E′.

[0170] S2. Select an edge with the smallest weight from the fully connected graph E and add it to E′.

[0171] S3. Take a node of the newly added edge in E′ as the starting node, search in E for an edge with the smallest weight connected to the starting node and which will not form a loop with the edges in E′, add it to E′, and repeat S3 until the number of nodes in E′ reaches the set total number, or until no edge that meets the conditions is found in E.

[0172] Among them, if there are multiple edges in S3 that have the smallest edge weight connected to the starting node and will not form a loop with the edges in E′, then you can search for edges that meet the user-set rules among the multiple edges, such as edges whose text length meets the requirements, and select one from the multiple edges that meet the user-set rules and add it to E′.

[0173] Further optionally, if no edge satisfying the user-set rules is found among the multiple edges, it means that the current newly added edge in E′ is not an optimal edge. The current newly added edge can be deleted from E′, and S3 is returned to be re-executed, and the search result is limited to not including the current newly added edge.

[0174] The texts corresponding to each node in E′ finally obtained constitute the target text set.

[0175] Reference Figure 6 , Figure 6 The flowchart of the difference text screening solution of this application is illustrated.

[0176] For the text set to be screened, which contains several texts. First, the similarity between each two texts in the text set to be screened is calculated by the similarity matching model. The similarity can be divided according to the level, and then a fully connected graph is constructed based on the similarity calculation results. The greedy algorithm is used to screen the difference texts from the fully connected graph, and finally the screened text set is obtained as a high-quality text set. This high-quality text set can be annotated by users, or directly used to train the neural network model for processing downstream tasks.

[0177] By adopting the solution of the present application, different data sets in the same field can be screened for differential texts based on the trained similarity matching model, which has stronger reusability and faster calculation speed.

[0178] The following is a description of a difference text screening device provided in an embodiment of the present application. The difference text screening device described below and the difference text screening method described above can be referenced to each other.

[0179] See also Figure 7 , Figure 7 A schematic diagram of the structure of a difference text screening device disclosed in an embodiment of the present application.

[0180] like Figure 7 As shown, the device may include:

[0181] A text set acquisition unit 11 is used to acquire a text set to be screened;

[0182] A similarity calculation unit 12, used for calculating the similarity between every two texts in the text set by using a preconfigured similarity matching model;

[0183] The similarity matching model is trained using training text pairs annotated with similarity labels as training data, wherein if the length difference between two training texts in a training text pair exceeds a set difference threshold, the similarity label of the training text pair is: the similarity with the largest median similarity between each candidate text after segmenting the longer training text and the shorter training text;

[0184] The text set screening unit 13 is used to screen out a target text set from the text set based on the similarity between any two texts in the text set, wherein the differences between the texts in the target text set meet a set difference condition.

[0185] Optionally, the apparatus of the present application may further include a model training data acquisition unit, which is used to acquire training data of the similarity matching model. The process may include:

[0186] Selecting a number of text pairs from a sample set, wherein the texts contained in the sample set and the text set to be screened belong to the same field;

[0187] Calculate the similarity labels for each of the text pairs:

[0188] If the length difference between the two texts in the text pair does not exceed the set difference threshold, the similarity between the two texts is calculated according to the set similarity calculation method as the similarity label of the text pair;

[0189] If the length difference between the two texts in the text pair exceeds a set difference threshold, the longer text of the two texts is segmented to obtain a number of candidate texts, and the similarity between each candidate text and the shorter text of the two texts is calculated according to the set similarity calculation method, and the similarity with the largest median value of the similarities between each candidate text and the shorter text is selected as the similarity label of the text pair;

[0190] Each of the text pairs and the corresponding similarity labels are used as training data for the similarity matching model.

[0191] The length difference between two texts in a text pair may be: the ratio of the length difference between the two texts to the sum of the lengths of the two texts.

[0192] Optionally, the process of the model training data acquisition unit calculating the similarity between two texts according to the set similarity calculation method may include:

[0193] Determine the embedding representation features of the two texts respectively, and calculate the first similarity between the two embedding representation features;

[0194] Determine the term frequency-inverse document frequency TF-IDF features of the two texts, and calculate the second similarity between the two TF-IDF features;

[0195] Calculate the third similarity based on the edit distance between the two texts;

[0196] The first similarity, the second similarity and the third similarity are weightedly added, and the result is used as the final similarity between the two texts.

[0197] Optionally, the process of the model training data acquisition unit determining the weights of the first similarity, the second similarity, and the third similarity may include:

[0198] Using a plurality of different weight combinations as weights of the first similarity, the second similarity and the third similarity, and calculating the similarities of the two text pairs in the sample set according to the set similarity calculation method under each weight combination;

[0199] Under each weight combination:

[0200] According to the calculated similarities of the two text pairs in the sample set, N texts whose differences meet the set difference conditions are selected from the sample set;

[0201] Projecting the sample set and the N texts into a two-dimensional space, and determining the text distribution density balance corresponding to the weight combination based on the distribution of the N texts and the sample set in the two-dimensional space;

[0202] A weight combination with the best text distribution density balance is selected as the final weight of each of the first similarity, the second similarity and the third similarity.

[0203] Optionally, the process in which the model training data acquisition unit projects the sample set and the N texts into a two-dimensional space, and determines the text distribution density balance corresponding to the weight combination based on the distribution of the N texts and the sample set in the two-dimensional space, may include:

[0204] Projecting each text in the sample set into a two-dimensional space to obtain an original sample space, and dividing the original sample space into a plurality of sub-areas;

[0205] Projecting the N texts into a two-dimensional space, and counting the number of texts projected into each of the sub-regions to obtain a text distribution density set;

[0206] The variance of the text distribution density set is calculated as the text distribution density balance corresponding to the weight combination, wherein the smaller the variance is, the better the text distribution density balance is.

[0207] Optionally, the similarity matching model may include a semantic representation acquisition module, a local representation acquisition module, a feature fusion module and a similarity prediction module. On this basis, the process of the similarity calculation unit calculating the similarity between each two texts in the text set using the preconfigured similarity matching model may include:

[0208] For every two texts in the text collection:

[0209] Calculate the semantic feature representation of each of the two texts through the semantic representation acquisition module;

[0210] Calculate the local feature representation between two texts through the local representation acquisition module;

[0211] Calculate the fused feature representation of the semantic feature representation of each of the two texts and the local feature representation through the feature fusion module;

[0212] The similarity prediction module predicts the similarity between two texts based on the fused feature representation.

[0213] Optionally, the semantic representation acquisition module may include a semantic representation extraction module and a first attention module, and the process in which the similarity calculation unit calculates the semantic feature representations of the two texts through the semantic representation acquisition module may include:

[0214] Obtaining primary semantic feature representations of the two texts respectively through the semantic representation extraction module;

[0215] Through the first attention module, a high-level semantic feature representation is determined based on the primary semantic feature representation of each text.

[0216] Optionally, the local representation acquisition module may include a sub-vector selection module and a second attention module, and the process in which the similarity calculation unit calculates the local feature representation between two texts through the local representation acquisition module may include:

[0217] Through the sub-vector selection module is the window size, slide on the primary semantic feature representation corresponding to the longer of the two texts to obtain a subvector, and multiply the obtained subvector with the primary semantic feature representation corresponding to the shorter of the two texts to obtain the vector multiplication result corresponding to each subvector; select the target subvector corresponding to the vector multiplication result of the maximum value; L is the length of the shorter text, To set parameters;

[0218] The local feature representation between the two texts is determined based on the target sub-vector through the second attention module.

[0219] Optionally, the process in which the similarity calculation unit calculates the fused feature representation of the semantic feature representations of the two texts and the local feature representations through the feature fusion module may include:

[0220] Through the feature fusion module, the first correlation feature and the first difference feature between the semantic feature representations of the two texts are calculated, and the second correlation feature and the second difference feature between the semantic feature representation of the shorter text of the two texts and the local feature representation are calculated; the semantic feature representations of the two texts, the first correlation feature, the first difference feature, the second correlation feature and / or the second difference feature are concatenated to obtain a fused feature representation.

[0221] Optionally, the process of the text set screening unit screening the target text set from the text set based on the similarity between any two texts in the text set may include:

[0222] Based on the similarity between two texts in the text set, a fully connected graph is constructed, wherein a node in the fully connected graph corresponds to a text in the text set, each node is connected to other nodes by an edge, and the weight of the edge is the similarity between the two texts corresponding to the two nodes connected by the edge;

[0223] According to the set selection rules, a path with the smallest total edge weight is selected from the fully connected graph, and the target text set is composed of the texts corresponding to the nodes on the selected path.

[0224] Optionally, the set selection rules include the total number of texts in the target text set and the length requirement of the texts in the target text set; when selecting the path with the smallest total edge weight from the fully connected graph, a greedy algorithm is used.

[0225] The difference text screening device provided in the embodiment of the present application can be applied to difference text screening devices, such as terminals: mobile phones, computers, etc. Optionally, Figure 8 The hardware structure diagram of the difference text screening device is shown in FIG. Figure 8 ,The hardware structure of the difference text screening device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0226] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0227] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0228] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0229] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0230] Get the text collection to be filtered;

[0231] Calculating the similarity between every two texts in the text collection using a preconfigured similarity matching model;

[0232] The similarity matching model is trained using training text pairs annotated with similarity labels as training data, wherein if the length difference between two training texts in a training text pair exceeds a set difference threshold, the similarity label of the training text pair is: the similarity with the largest median similarity between each candidate text after segmenting the longer training text and the shorter training text;

[0233] Based on the similarity between any two texts in the text set, a target text set is screened out from the text set, and the differences between the texts in the target text set meet a set difference condition.

[0234] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0235] The embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0236] Get the text collection to be filtered;

[0237] Calculating the similarity between every two texts in the text collection using a preconfigured similarity matching model;

[0238] The similarity matching model is trained using training text pairs annotated with similarity labels as training data, wherein if the length difference between two training texts in a training text pair exceeds a set difference threshold, the similarity label of the training text pair is: the similarity with the largest median similarity between each candidate text after segmenting the longer training text and the shorter training text;

[0239] Based on the similarity between any two texts in the text set, a target text set is screened out from the text set, and the differences between the texts in the target text set meet a set difference condition.

[0240] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0241] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0242] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0243] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for screening difference texts, characterized in that: include: Get the text collection to be filtered; Calculating the first similarity, the second similarity and the third similarity between every two texts in the text set by using a preconfigured similarity matching model, performing weighted addition on the first similarity, the second similarity and the third similarity, and using the result as the final similarity between the two texts; The process of determining the weights of the first similarity, the second similarity and the third similarity includes: using a plurality of different weight combinations as the weights of the first similarity, the second similarity and the third similarity, and under each weight combination, calculating the similarity of two pairs of texts in a sample set according to a set similarity calculation method, wherein the texts contained in the sample set and the text set to be screened belong to the same field; under each weight combination: selecting N texts whose differences meet the set difference conditions from the sample set according to the calculated similarities of two pairs of texts in the sample set; projecting each text in the sample set into a two-dimensional space to obtain an original sample space, and dividing the original sample space into a plurality of sub-areas; projecting the N texts into a two-dimensional space, and counting the number of texts projected into each of the sub-areas to obtain a text distribution density set; calculating the variance of the text distribution density set as the text distribution density balance corresponding to the weight combination; selecting the weight combination with the best text distribution density balance as the final weights of the first similarity, the second similarity and the third similarity; The similarity matching model is trained using training text pairs annotated with similarity labels as training data, wherein if the length difference between two training texts in a training text pair exceeds a set difference threshold, the similarity label of the training text pair is: the similarity with the largest median similarity between each candidate text after segmenting the longer training text and the shorter training text; Based on the final similarities between any two texts in the text set, a target text set is screened out from the text set, and the differences between the texts in the target text set meet the set difference conditions.

2. The method according to claim 1, characterized in that The process of acquiring the training data of the similarity matching model includes: Select several text pairs from the sample set; Calculate the similarity labels for each of the text pairs: If the length difference between the two texts in the text pair does not exceed the set difference threshold, the similarity between the two texts is calculated according to the set similarity calculation method as the similarity label of the text pair; If the length difference between the two texts in the text pair exceeds a set difference threshold, the longer text of the two texts is segmented to obtain a number of candidate texts, and the similarity between each candidate text and the shorter text of the two texts is calculated according to the set similarity calculation method, and the similarity with the largest median value of the similarities between each candidate text and the shorter text is selected as the similarity label of the text pair; Each of the text pairs and the corresponding similarity labels are used as training data for the similarity matching model.

3. The method according to claim 2, characterized in that The length difference of two texts in a text pair is the ratio of the length difference of the two texts to the sum of the lengths of the two texts.

4. The method according to claim 2, characterized in that: The process of calculating the similarity between two texts according to the set similarity calculation method includes: Determine the embedding representation features of the two texts respectively, and calculate the first similarity between the two embedding representation features; Determine the term frequency-inverse document frequency TF-IDF features of the two texts, and calculate the second similarity between the two TF-IDF features; Calculate the third similarity based on the edit distance between the two texts; The first similarity, the second similarity and the third similarity are weightedly added, and the result is used as the final similarity between the two texts.

5. The method according to any one of claims 1 to 4, characterized in that: The similarity matching model includes a semantic representation acquisition module, a local representation acquisition module, a feature fusion module and a similarity prediction module; The calculating the similarity between every two texts in the text set by using a preconfigured similarity matching model includes: For every two texts in the text collection: Calculate the semantic feature representation of each of the two texts through the semantic representation acquisition module; Calculate the local feature representation between two texts through the local representation acquisition module; Calculate the fused feature representation of the semantic feature representation of each of the two texts and the local feature representation through the feature fusion module; The similarity prediction module predicts the similarity between two texts based on the fused feature representation.

6. The method according to claim 5, characterized in that The semantic representation acquisition module includes a semantic representation extraction module and a first attention module, and the semantic feature representations of the two texts are calculated by the semantic representation acquisition module, including: Obtaining primary semantic feature representations of the two texts respectively through the semantic representation extraction module; Through the first attention module, a high-level semantic feature representation is determined based on the primary semantic feature representation of each text.

7. The method according to claim 6, characterized in that The local representation acquisition module includes a sub-vector selection module and a second attention module, and the local feature representation between two texts is calculated by the local representation acquisition module, including: Through the sub-vector selection module is the window size, slide on the primary semantic feature representation corresponding to the longer of the two texts to obtain a subvector, and multiply the obtained subvector with the primary semantic feature representation corresponding to the shorter of the two texts to obtain the vector multiplication result corresponding to each subvector; select the target subvector corresponding to the vector multiplication result of the maximum value; L is the length of the shorter text, To set parameters; The local feature representation between the two texts is determined based on the target sub-vector through the second attention module.

8. The method according to claim 5, characterized in that Calculating the fused feature representation of the semantic feature representations of the two texts and the local feature representations by the feature fusion module includes: Through the feature fusion module, the first correlation feature and the first difference feature between the semantic feature representations of the two texts are calculated, and the second correlation feature and the second difference feature between the semantic feature representation of the shorter text of the two texts and the local feature representation are calculated; the semantic feature representations of the two texts, the first correlation feature, the first difference feature, the second correlation feature and / or the second difference feature are concatenated to obtain a fused feature representation.

9. The method according to any one of claims 1 to 4, characterized in that: Based on the similarity between two texts in the text set, a target text set is screened out from the text set, including: Based on the similarity between two texts in the text set, a fully connected graph is constructed, wherein a node in the fully connected graph corresponds to a text in the text set, each node is connected to other nodes by an edge, and the weight of the edge is the similarity between the two texts corresponding to the two nodes connected by the edge; According to the set selection rules, a path with the smallest total edge weight is selected from the fully connected graph, and the target text set is composed of the texts corresponding to the nodes on the selected path.

10. The method according to claim 9, characterized in that The set selection rules include the total number of texts in the target text set and the length requirement of the texts in the target text set; When selecting a path with the smallest total edge weight from the fully connected graph, a greedy algorithm is used.

11. A device for screening difference texts, characterized in that: include: A text collection acquisition unit, used to acquire a text collection to be screened; A similarity calculation unit, used for calculating the first similarity, the second similarity and the third similarity between every two texts in the text set by using a preconfigured similarity matching model, performing weighted addition on the first similarity, the second similarity and the third similarity, and taking the result as the final similarity between the two texts; The process of determining the weights of the first similarity, the second similarity and the third similarity includes: using a plurality of different weight combinations as the weights of the first similarity, the second similarity and the third similarity, and under each weight combination, calculating the similarity of two pairs of texts in a sample set according to a set similarity calculation method, wherein the texts contained in the sample set and the text set to be screened belong to the same field; under each weight combination: selecting N texts whose differences meet the set difference conditions from the sample set according to the calculated similarities of two pairs of texts in the sample set; projecting each text in the sample set into a two-dimensional space to obtain an original sample space, and dividing the original sample space into a plurality of sub-areas; projecting the N texts into a two-dimensional space, and counting the number of texts projected into each of the sub-areas to obtain a text distribution density set; calculating the variance of the text distribution density set as the text distribution density balance corresponding to the weight combination; selecting the weight combination with the best text distribution density balance as the final weights of the first similarity, the second similarity and the third similarity; The similarity matching model is trained using training text pairs annotated with similarity labels as training data, wherein if the length difference between two training texts in a training text pair exceeds a set difference threshold, the similarity label of the training text pair is: the similarity with the largest median similarity between each candidate text after segmenting the longer training text and the shorter training text; The text set screening unit is used to screen out a target text set from the text set based on the final similarity between two texts in the text set, wherein the differences between the texts in the target text set meet the set difference conditions.

12. A differential text screening device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the difference text screening method as described in any one of claims 1 to 10.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the difference text screening method as described in any one of claims 1 to 10 is implemented.

Images