Semantic similarity model training set acquisition method and apparatus, and computer device

Abstract

The application provides a method and device for obtaining a training set of a semantic similarity model and a computer device, the method comprising: grouping similar texts in an original corpus to construct a plurality of similar training samples; for each first natural language text, determining target language texts that are not similar in semantics to the first natural language text in the original corpus according to the similar text group to which the first natural language text belongs, and calculating the literal similarity between the first natural language text and each target language text, and constructing M dissimilar training samples based on the M target language texts with the highest literal similarity; and using each similar training sample and each dissimilar training sample as a training set for obtaining a semantic similarity model. The application can improve the generalization and detection accuracy of the semantic similarity model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, storage medium and computer device for obtaining training sets of a semantic similarity model. Background Technology

[0002] Semantic similarity models are used to detect whether multiple natural language texts with different expressions convey similar meanings. In obtaining a semantic similarity model, a training set needs to be constructed beforehand. This training set includes multiple sets of semantic training data, each set containing two natural language texts with different expressions and annotations indicating whether the two texts have the same semantic meaning. After the training set is constructed, it can be used to train an initial model to obtain the semantic similarity model. However, in practical applications, it has been found that current semantic similarity models suffer from low detection accuracy. Summary of the Invention

[0003] The purpose of this application is to address at least one of the aforementioned technical deficiencies, particularly the low detection accuracy in the prior art.

[0004] Firstly, this application provides a method for obtaining a training set for a semantic similarity model, the method comprising:

[0005] Based on the various similar text groups in the original corpus, multiple similar training samples are constructed; wherein, each of the similar text groups includes multiple natural language texts that are semantically similar in pairs, and each similar training sample includes a first natural language text and a second natural language text belonging to the same similar text group;

[0006] For each first natural language text, based on the similar text group to which the first natural language text belongs, each target language text that is semantically dissimilar to the first natural language text is determined in the original corpus, and the literal similarity between the first natural language text and each target language text is calculated respectively. Based on the M target language texts with the highest literal similarity, M dissimilar training samples are constructed; where M is a preset positive integer.

[0007] The similar training samples and the dissimilar training samples are used as the training set for obtaining the semantic similarity model.

[0008] In one embodiment, before the step of using the similar training samples and the dissimilar training samples as the training set for obtaining the semantic similarity model, the method further includes:

[0009] Obtain a preset number of dissimilar training samples, wherein the number of dissimilar training samples is determined based on a preset sample ratio;

[0010] For each first natural language text, (NM) target language texts are randomly selected from each of the target language texts corresponding to the first natural language text, and (NM) dissimilar training samples are constructed based on the randomly selected (NM) target language texts; where N is the number of dissimilar training samples, and N is greater than M.

[0011] In one embodiment, the step of randomly selecting (NM) target language texts from each of the target language texts corresponding to the first natural language text includes:

[0012] Randomly select multiple target language texts from the target language texts corresponding to the first natural language text;

[0013] Based on the M target language texts with the highest literal similarity, multiple randomly selected target language texts are deduplicated to obtain (NM) target language texts.

[0014] In one embodiment, the step of determining, based on the similar text grouping to which the first natural language text belongs, various target language texts that are semantically dissimilar to the first natural language text in the original corpus includes:

[0015] Each natural language text in the original corpus that belongs to a different similar text group than the first natural language text is taken as a target language text that is semantically dissimilar to the first natural language text.

[0016] In one embodiment, the step of constructing multiple similar training samples based on various similar text groups in the original corpus includes:

[0017] For each of the similar text groups, the natural language texts belonging to that similar text group are combined in pairs to obtain multiple similar training samples.

[0018] In one embodiment, the step of calculating the literal similarity between the first natural language text and each of the target language texts includes:

[0019] The edit distance between the first natural language text and each of the target language texts is calculated, and the edit distance is used to reflect literal similarity.

[0020] Secondly, this application provides a training set acquisition device for a semantic similarity model, the device comprising:

[0021] The similar training sample construction module is used to construct multiple similar training samples based on the various similar text groups in the original corpus; wherein, each similar text group includes multiple natural language texts that are semantically similar in pairs, and each similar training sample includes a first natural language text and a second natural language text belonging to the same similar text group;

[0022] The first dissimilar training sample construction module is used to, for each first natural language text, determine each target language text in the original corpus that is semantically dissimilar to the first natural language text according to the similar text group to which the first natural language text belongs, calculate the literal similarity between the first natural language text and each target language text respectively, and construct M dissimilar training samples based on the M target language texts with the highest literal similarity; where M is a preset positive integer;

[0023] The training set acquisition module is used to use the various similar training samples and the various dissimilar training samples as the training set for acquiring the semantic similarity model.

[0024] In one embodiment, the training set acquisition device for the semantic similarity model further includes:

[0025] The quantity acquisition module is used to acquire a preset number of dissimilar training samples, wherein the number of dissimilar training samples is determined according to a preset sample ratio;

[0026] The second dissimilar training sample construction module is used to, for each first natural language text, randomly select (NM) target language texts from each target language text corresponding to the first natural language text, and construct (NM) dissimilar training samples based on the randomly selected (NM) target language texts; wherein, N is the number of dissimilar training samples, and N is greater than M.

[0027] Thirdly, embodiments of this application also provide a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the training set acquisition method for the semantic similarity model described in any of the above embodiments.

[0028] Fourthly, embodiments of this application also provide a computer device, which includes: one or more processors, and a memory;

[0029] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the training set acquisition method for the semantic similarity model described in any of the above embodiments.

[0030] In the training set acquisition method, apparatus, storage medium, and computer equipment of the semantic similarity model in this application, multiple similar training samples can be constructed based on the similar text groups in the original corpus. Each similar training sample includes a first natural language text and a second natural language text that are semantically similar. For the first natural language text in each similar training sample, this application can determine each target language text that is semantically dissimilar to the first natural language text in the original corpus according to the similar text group to which the first natural language text belongs, and calculate the literal similarity between the first natural language text and each target language text respectively. Based on the M target language texts with the highest literal similarity, M dissimilar training samples are constructed, such that the M dissimilar training samples all include natural language texts that are literally similar but semantically dissimilar. The semantic similarity model is trained using each dissimilar training sample and each similar training sample as the training set, so that the trained semantic similarity model can accurately detect cases that are literally similar but semantically dissimilar, thereby improving the generalization and detection accuracy of the semantic similarity model. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is one of the flowcharts illustrating the method for obtaining the training set of a semantic similarity model in one embodiment;

[0033] Figure 2 This is the second flowchart illustrating the method for obtaining the training set of a semantic similarity model in one embodiment;

[0034] Figure 3 This is a schematic diagram of the structure of a training set acquisition device for a semantic similarity model in one embodiment;

[0035] Figure 4 This is a schematic diagram of the structure of a computer device in one embodiment. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] As mentioned in the background section, current semantic similarity models suffer from low detection accuracy. The inventors' research revealed that this problem stems from the fact that, when constructing the training set, the number of similar samples for each natural language text is limited, while the number of dissimilar samples is enormous. Therefore, constructing the training set typically faces a severe data imbalance problem. To alleviate this issue, the ratio of similar to dissimilar samples is usually controlled within a certain range, such as 1:5 or 1:10. To construct a training set that meets the preset ratio, existing techniques randomly select a preset number of dissimilar samples from the original corpus and add these selected dissimilar samples to the training set. However, the semantic dissimilarity representations of each natural language text are rich and diverse. In the original corpus, the same natural language text corresponds to multiple dissimilar samples with low literal similarity and multiple dissimilar samples with high literal similarity. Moreover, the number of dissimilar samples with low literal similarity is far greater than the number of dissimilar samples with high literal similarity. Therefore, in a predetermined number of randomly selected dissimilar samples, the proportion of dissimilar samples with low literal similarity is extremely high. Using this training set to train the initial model makes the model's learning task too simple, resulting in a semantic similarity model that can only distinguish between texts that are obviously dissimilar in literal sense and semantically dissimilar, but cannot correctly predict text pairs that are literally similar but semantically dissimilar, thus reducing the detection accuracy of the semantic similarity model.

[0038] To address the aforementioned issues, this application provides a method, apparatus, storage medium, and computer device for acquiring a training set for a semantic similarity model. By constructing multiple dissimilar samples that are literally similar but semantically dissimilar, and using these samples to train a semantic similarity model, the trained semantic similarity model can accurately detect cases where the samples are literally similar but semantically dissimilar, thereby improving the generalization and detection accuracy of the semantic similarity model.

[0039] The method for obtaining the training set of the semantic similarity model provided in this application can be applied to computer devices. It is understood that the computer devices mentioned herein can be devices with data processing capabilities, and can be, but are not limited to, a single server, a server cluster consisting of multiple servers, a personal laptop or desktop computer, etc.

[0040] Example 1:

[0041] like Figure 1 As shown, the method for obtaining the training set of the semantic similarity model in this application may include the following steps:

[0042] S102: Construct multiple similar training samples based on the various similar text groups in the original corpus; wherein, each of the similar text groups includes multiple natural language texts that are semantically similar in pairs, and each similar training sample includes a first natural language text and a second natural language text belonging to the same similar text group.

[0043] Specifically, the original corpus can be a corpus used to store various natural language texts. When constructing the training set, multiple natural language texts can be extracted from the original corpus, and training samples can be constructed accordingly. In the original corpus, the natural language texts have already been grouped into similar text groups. Semantically similar natural language texts belong to the same similar text group, while semantically dissimilar natural language texts belong to different similar text groups. In one example, each similar text group includes a standard sentence and multiple similar sentences that are semantically similar to that standard sentence. The standard sentences included in different similar text groups are semantically dissimilar. For example, in the original corpus shown in Table 1, "I want to return the goods," "I want to return the goods," and "want to return the goods" belong to the same similar text group, while "The item sent is damaged," "The item sent by the courier is damaged," and "The item sent is damaged" belong to another similar text group.

[0044] Table 1 Examples of similar text grouping in the original corpus

[0045] Standard Sentence Similar sentences I want to return the item. I want to return the item. I want to return the goods. Want to return the goods The item that was sent was damaged. The item that was sent was damaged. The package that was delivered by courier was damaged. The item that was sent was damaged.

[0046] The computer device can construct multiple similar training samples based on the various similar text groups in the original corpus, such that each similar training sample includes a first natural language text and a second natural language text belonging to the same similar text group. Furthermore, each similar training sample may also include similarity annotation results, which are used to reflect that the two natural language texts in the training sample are semantically similar. It is understood that the number of constructed similar training samples can be determined according to actual circumstances, such as based on the amount of natural language text data in the original corpus and / or a preset number of similar samples; this application does not impose specific limitations in this regard.

[0047] In one embodiment, the step of constructing multiple similar training samples based on the various similar text groups in the original corpus may include: for each similar text group, pairwise combining the natural language texts belonging to that similar text group to obtain multiple similar training samples. For example, in the example shown in Table 1, for the similar text group to which "I want to return the goods" belongs, the computer device can construct three similar training samples based on that similar text group: "I want to return the goods" and "I want to return the goods", "I want to return the goods" and "I want to return the goods" and "I want to return the goods" and "I want to return the goods". Each similar text group constructs similar training samples according to the aforementioned process, thereby obtaining multiple similar training samples. In this way, by pairwise combining all natural language texts belonging to the same similar text group to obtain multiple similar training samples, the similar training samples can be made diverse. Using these similar training samples to obtain a semantic similarity model, the semantic similarity model can more accurately detect whether the input natural language texts are semantically similar, thereby further improving the detection accuracy of the model.

[0048] S104: For each first natural language text, based on the similar text group to which the first natural language text belongs, determine each target language text that is semantically dissimilar to the first natural language text in the original corpus, and calculate the literal similarity between the first natural language text and each target language text respectively. Based on the M target language texts with the highest literal similarity, construct M dissimilar training samples; where M is a preset positive integer.

[0049] Literal similarity refers to the degree of similarity between two natural language texts on a literal level. It can be determined based on factors such as the words included in the two natural language texts and / or the order of those words. It should be noted that high literal similarity does not necessarily equate to high semantic similarity. Two natural language texts with high literal similarity can be semantically similar or semantically dissimilar.

[0050] For the first natural language text in each similar training sample, the computer device can perform the following steps A1 to A5:

[0051] Step A1: Based on the similar text group to which the first natural language text belongs, identify each target language text in the original corpus that is semantically dissimilar to the first natural language text. That is, each target language text is semantically dissimilar to the first natural language text. In one embodiment, the computer device can use each natural language text in the original corpus that belongs to a different similar text group than the first natural language text as the target language texts that are semantically dissimilar to the first natural language text. In other words, if the similar text group to which the first natural language text belongs is the first group, the computer device can use the remaining groups other than the first group as the target groups, and use all the natural language texts of each target group as the target language texts corresponding to the first natural language text. This facilitates the subsequent selection of target language texts with higher literal similarity but semantic dissimilarity, and the construction of dissimilar training samples based on these samples to train the semantic similarity model, thereby further improving the detection accuracy of the model.

[0052] Step A3: Calculate the literal similarity between the first natural language text and each corresponding target language text. A higher literal similarity indicates a greater literal similarity between the first natural language text and its corresponding target language text. It is understood that this application can use any method in the prior art to calculate the literal similarity between two natural language texts. In one example, this application can calculate the edit distance between the first natural language text and each target language text separately to reflect the literal similarity. This simplifies the calculation of literal similarity and shortens the time required to obtain the training set.

[0053] Step A5: Construct M dissimilar training samples based on the M target language texts with the highest literal similarity. That is, construct M dissimilar training samples based on the target language texts corresponding to the M literal similarities from highest to lowest, ensuring that each of the M dissimilar training samples includes natural language texts that are literally similar but semantically dissimilar. Further, the Mth dissimilar training sample may include the first natural language text, the Mth target language text, and the dissimilarity annotation result, which reflects that the two natural language texts in the training sample are semantically dissimilar.

[0054] For example, when M is 3, the computer device can construct three dissimilar training samples based on the target language text with the highest numerical literal similarity (i.e., the highest literal similarity), the target language text with the second highest numerical literal similarity, and the target language text with the third highest numerical literal similarity.

[0055] S106: Use each of the similar training samples and each of the dissimilar training samples as a training set for obtaining the semantic similarity model.

[0056] Specifically, the computer device can construct a training set by using similar training samples as positive examples and dissimilar training samples as negative examples. This training set can be used to train the initial semantic similarity model and obtain the trained semantic similarity model. In one example, the training set data is shown in Table 2:

[0057] Table 2 Examples of Training Sets

[0058]

[0059]

[0060] This application constructs multiple similar training samples based on various similar text groups in the original corpus. Each similar training sample includes semantically similar first and second natural language texts. For the first natural language text in each similar training sample, this application identifies semantically dissimilar target language texts in the original corpus according to the similar text group to which the first natural language text belongs. The literal similarity between the first natural language text and each target language text is calculated. M dissimilar training samples are constructed based on the M target language texts with the highest literal similarity, ensuring that all M dissimilar training samples include literally similar but semantically dissimilar natural language texts. The semantic similarity model is trained using each dissimilar training sample and each similar training sample as the training set. This allows the trained semantic similarity model to accurately detect cases of literal similarity but semantic dissimilarity, thereby improving the generalization and detection accuracy of the semantic similarity model.

[0061] Example 2:

[0062] like Figure 2 As shown, the method for obtaining the training set of the semantic similarity model in this application may include the following steps:

[0063] S202: Construct multiple similar training samples based on the various similar text groups in the original corpus; wherein, each of the similar text groups includes multiple natural language texts that are semantically similar in pairs, and each similar training sample includes a first natural language text and a second natural language text belonging to the same similar text group.

[0064] For the specific limitations of S202 in this step, please refer to the relevant description of S102, which will not be repeated here.

[0065] S204: Obtain a preset number of dissimilar training samples, wherein the number of dissimilar training samples is determined according to a preset sample ratio.

[0066] The number of dissimilar training samples can be pre-set and conform to a preset sample ratio. Specifically, since the number of semantically similar samples for each natural language text is limited and small, while the number of semantically dissimilar samples for each natural language text is large, to alleviate the problem of imbalanced sample data, a training set can be constructed based on this number of dissimilar training samples, so that the ratio between similar and dissimilar training samples in the training set is controlled at the preset sample ratio, such as 1:5 or 1:10.

[0067] S206: For each of the first natural language texts, based on the similar text group to which the first natural language text belongs, determine each target language text in the original corpus that is semantically dissimilar to the first natural language text, and calculate the literal similarity between the first natural language text and each of the target language texts respectively. Construct M dissimilar training samples based on the M target language texts with the highest literal similarity; where M is a preset positive integer.

[0068] For the specific limitations of S206 in this step, please refer to the relevant description of S104, which will not be repeated here.

[0069] S208: For each first natural language text, randomly select (NM) target language texts from each target language text corresponding to the first natural language text, and construct (NM) dissimilar training samples based on the randomly selected (NM) target language texts; where N is the number of dissimilar training samples, and N is greater than M.

[0070] Specifically, for each similar training sample containing a first natural language text, after determining the corresponding target language texts, the computer device can randomly select (NM) target language texts and construct (NM) dissimilar training samples accordingly. Each of these (NM) dissimilar training samples can include two natural language texts that are semantically and literally dissimilar. In this way, on the one hand, the computer device can construct N dissimilar training samples based on each similar training sample, ensuring that the ratio of similar to dissimilar training samples in the training set is controlled within a preset ratio, avoiding data imbalance. On the other hand, the constructed dissimilar training samples include both literally similar but semantically dissimilar samples and literally dissimilar and semantically dissimilar samples. Training a semantic similarity model based on this allows the trained semantic similarity model to more accurately identify whether the input natural language texts are semantically similar, thereby further improving detection accuracy.

[0071] In one example, N can be equal to 2M. In this case, of the N dissimilar training samples constructed based on the same similar training samples, half are constructed based on literal similarity, and the other half are constructed based on random selection, in order to further improve the accuracy of model detection.

[0072] In one embodiment, the step of randomly selecting (NM) target language texts from each of the target language texts corresponding to the first natural language text includes:

[0073] Randomly select multiple target language texts from the target language texts corresponding to the first natural language text;

[0074] Based on the M target language texts with the highest literal similarity, multiple randomly selected target language texts are deduplicated to obtain (NM) target language texts.

[0075] Specifically, when constructing N dissimilar training samples based on each similar training sample, to avoid duplicate samples among the N dissimilar training samples constructed from the same similar training sample, when constructing (NM) dissimilar training samples based on randomly selected target language text, the computer device can perform deduplication checks on the randomly selected target language text and the target language text selected based on literal similarity, and then deduplicate each of the randomly selected target language texts. This ensures that after deduplication, there is no overlap between the (NM) randomly selected target language texts and the M target language texts selected based on literal similarity. The computer device can then construct (NM) dissimilar training samples based on the deduplicated (NM) target language texts, ensuring that there are no duplicate samples among the N dissimilar training samples constructed from the same similar training sample, thereby further improving the accuracy of model detection.

[0076] S210: Use each of the similar training samples and each of the dissimilar training samples as a training set for obtaining the semantic similarity model.

[0077] Specifically, the computer device can construct a training set by using multiple similar training samples as positive examples and N dissimilar training samples corresponding to each similar training sample as negative examples. This training set can be used to train the initial semantic similarity model and obtain the trained semantic similarity model.

[0078] The following describes the training set acquisition device for the semantic similarity model provided in the embodiments of this application. The training set acquisition device for the semantic similarity model described below and the training set acquisition method for the semantic similarity model described above can be referred to in correspondence.

[0079] In one embodiment, this application provides a training set acquisition device 300 for a semantic similarity model. For example... Figure 3 As shown, the device 300 includes a similar training sample construction module 310, a first dissimilar training sample construction module 320, and a training set acquisition module 330. Wherein:

[0080] The similar training sample construction module 310 is used to construct multiple similar training samples based on the various similar text groups in the original corpus; wherein, each similar text group includes multiple natural language texts that are semantically similar in pairs, and each similar training sample includes a first natural language text and a second natural language text belonging to the same similar text group;

[0081] The first dissimilar training sample construction module 320 is used to, for each first natural language text, determine each target language text in the original corpus that is semantically dissimilar to the first natural language text according to the similar text group to which the first natural language text belongs, calculate the literal similarity between the first natural language text and each target language text respectively, and construct M dissimilar training samples based on the M target language texts with the highest literal similarity; where M is a preset positive integer;

[0082] The training set acquisition module 330 is used to use each of the similar training samples and each of the dissimilar training samples as a training set for acquiring the semantic similarity model.

[0083] In one embodiment, the training set acquisition device 300 for the semantic similarity model of this application further includes a quantity acquisition module and a second dissimilar training sample construction module. Wherein:

[0084] The quantity acquisition module is used to acquire a preset number of dissimilar training samples, wherein the number of dissimilar training samples is determined according to a preset sample ratio;

[0085] The second dissimilar training sample construction module is used to, for each first natural language text, randomly select (NM) target language texts from each target language text corresponding to the first natural language text, and construct (NM) dissimilar training samples based on the randomly selected (NM) target language texts; wherein, N is the number of dissimilar training samples, and N is greater than M.

[0086] In one embodiment, the second dissimilar training sample construction module includes a random selection unit and a deduplication unit. The random selection unit is used to randomly select multiple target language texts from each of the target language texts corresponding to the first natural language text. The deduplication unit is used to deduplicatively select multiple target language texts based on the M target language texts with the highest literal similarity, to obtain (NM) target language texts.

[0087] In one embodiment, the first dissimilar training sample construction module 320 includes a target language text determination unit. This target language text determination unit is used to identify each natural language text in the original corpus that belongs to a different similar text group than the first natural language text as a target language text that is semantically dissimilar to the first natural language text.

[0088] In one embodiment, the similar training sample construction module 310 includes a combination unit. This combination unit is used to combine each natural language text belonging to each of the similar text groups in pairs to obtain multiple similar training samples.

[0089] In one embodiment, the first dissimilar training sample construction module 320 includes a literal similarity calculation unit. This literal similarity calculation unit is used to calculate the edit distance between the first natural language text and each of the target language texts, respectively, the edit distance reflecting the literal similarity.

[0090] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the training set acquisition method for the semantic similarity model as described in any of the above embodiments.

[0091] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the training set acquisition method for the semantic similarity model as described in any of the above embodiments.

[0092] Indicatively, Figure 4 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. (Refer to...) Figure 4The computer device 900 includes a processing component 902, which further includes one or more processors, and memory resources represented by memory 901 for storing instructions, such as application programs, that can be executed by the processing component 902. The application programs stored in memory 901 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 902 is configured to execute instructions to perform the steps of the commodity attribute detection method described in any of the above embodiments.

[0093] The computer device 900 may also include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. The computer device 900 may operate on an operating system stored in memory 901, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0094] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0095] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the related listed items.

[0096] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0097] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for obtaining a training set for a semantic similarity model, characterized in that, The method includes: Based on the various similar text groups in the original corpus, multiple similar training samples are constructed; wherein, each of the similar text groups includes multiple natural language texts that are semantically similar in pairs, and each similar training sample includes a first natural language text and a second natural language text belonging to the same similar text group; For each first natural language text, based on the similar text group to which the first natural language text belongs, each target language text that is semantically dissimilar to the first natural language text is determined in the original corpus, and the literal similarity between the first natural language text and each target language text is calculated respectively. Based on the M target language texts with the highest literal similarity, M dissimilar training samples are constructed; where M is a preset positive integer. Obtain a preset number of dissimilar training samples, wherein the number of dissimilar training samples is determined based on a preset sample ratio; For each first natural language text, NM target language texts are randomly selected from each of the corresponding target language texts, and NM dissimilar training samples are constructed based on the randomly selected NM target language texts; where N is the number of dissimilar training samples, and N is greater than M; The similar training samples and the dissimilar training samples are used as the training set for obtaining the semantic similarity model; The step of randomly selecting NM target language texts from each of the target language texts corresponding to the first natural language text includes: Randomly select multiple target language texts from the target language texts corresponding to the first natural language text; Based on the M target language texts with the highest literal similarity, multiple randomly selected target language texts are deduplicated to obtain NM target language texts.

2. The method for obtaining the training set of the semantic similarity model according to claim 1, characterized in that, The step of determining, based on the similar text grouping to which the first natural language text belongs, each target language text in the original corpus that is semantically dissimilar to the first natural language text includes: Each natural language text in the original corpus that belongs to a different similar text group than the first natural language text is taken as a target language text that is semantically dissimilar to the first natural language text.

3. The method for obtaining the training set of the semantic similarity model according to claim 1 or 2, characterized in that, The step of constructing multiple similar training samples based on the grouping of similar texts in the original corpus includes: For each of the similar text groups, the natural language texts belonging to that similar text group are combined in pairs to obtain multiple similar training samples.

4. The method for obtaining the training set of the semantic similarity model according to claim 1 or 2, characterized in that, The step of calculating the literal similarity between the first natural language text and each of the target language texts includes: The edit distance between the first natural language text and each of the target language texts is calculated, and the edit distance is used to reflect literal similarity.

5. A device for acquiring a training set for a semantic similarity model, characterized in that, The device includes: The similar training sample construction module is used to construct multiple similar training samples based on the various similar text groups in the original corpus; wherein, each similar text group includes multiple natural language texts that are semantically similar in pairs, and each similar training sample includes a first natural language text and a second natural language text belonging to the same similar text group; The first dissimilar training sample construction module is used to, for each first natural language text, determine each target language text in the original corpus that is semantically dissimilar to the first natural language text according to the similar text group to which the first natural language text belongs, calculate the literal similarity between the first natural language text and each target language text respectively, and construct M dissimilar training samples based on the M target language texts with the highest literal similarity; where M is a preset positive integer; The quantity acquisition module is used to acquire a preset number of dissimilar training samples, wherein the number of dissimilar training samples is determined according to a preset sample ratio; The second dissimilar training sample construction module is used to, for each first natural language text, randomly select NM target language texts from each target language text corresponding to the first natural language text, and construct NM dissimilar training samples based on the randomly selected NM target language texts; where N is the number of dissimilar training samples, and N is greater than M; The training set acquisition module is used to use each of the similar training samples and each of the dissimilar training samples as the training set for acquiring the semantic similarity model; The second dissimilar training sample construction module includes: The random selection unit is used to randomly select multiple target language texts from each of the target language texts corresponding to the first natural language text; The deduplication unit is used to deduplicatize multiple randomly selected target language texts based on the M target language texts with the highest literal similarity, so as to obtain NM target language texts.

6. A storage medium, characterized in that: The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the training set acquisition method for the semantic similarity model as described in any one of claims 1 to 4.

7. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the training set acquisition method for the semantic similarity model as described in any one of claims 1 to 4.

Images