Text similarity calculation method for searching scene and double-tower model training method

Abstract

The disclosure discloses a text similarity calculation method for searching a scene and a double-tower model training method, relates to the technical field of computers, further relates to the fields of deep learning and natural language processing. The specific implementation scheme is as follows: a first query text in a search request is acquired, and a first resource text of a to-be-processed resource is acquired; wherein the to-be-processed resource is a resource recalled based on the search request; prompt information is added in the first query text and the first resource text respectively, so that a second query text and a second resource text are obtained; semantic information extraction is respectively performed on the second query text and the second resource text, so that a first semantic vector of the second query text and a second semantic vector of the second resource text are obtained; and similarity between the query text and the resource text is calculated according to the first semantic vector and the second semantic vector. The semantic loss problem caused by the difference in the data structure of the model input can be solved to a certain extent, and the model performance is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and further to the fields of deep learning and natural language processing, and particularly to a text similarity calculation method and a dual-tower model training method for search scenarios. Background Technology

[0002] In search scenarios, when users enter query text, it is often necessary to display search results with higher relevance to them. For example, in advertising search scenarios, relevance models are used to obtain the relevance between query text and candidate search results, thereby filtering out high-quality advertising content that meets user needs. Therefore, improving the accuracy of similarity calculation between texts has become one of the important research directions in this field. Summary of the Invention

[0003] This disclosure provides a method for calculating text similarity and a dual-tower model training method for search scenarios.

[0004] According to a first aspect of this disclosure, a method for calculating text similarity in a search scenario is provided, comprising:

[0005] Obtain the first query text from the search request and obtain the first resource text of the resource to be processed; wherein the resource to be processed is the resource recalled based on the search request;

[0006] Add prompt information to the first query text and the first resource text respectively to obtain the second query text and the second resource text;

[0007] Semantic information is extracted from the second query text and the second resource text respectively to obtain the first semantic vector of the second query text and the second semantic vector of the second resource text.

[0008] The similarity between the query text and the resource text is calculated based on the first semantic vector and the second semantic vector.

[0009] According to a second aspect of this disclosure, a dual-tower model training method is provided, wherein the dual-tower model is used to compute text semantic similarity in a search scenario, the method comprising:

[0010] Acquire training data, which includes a first text sample pair and a label for the first text sample pair; wherein the first text sample pair includes a first text and a second text;

[0011] Add a prompt message to the first text to obtain the third text, and add a prompt message to the second text to obtain the fourth text;

[0012] The third and fourth texts are respectively tokenized to obtain the model input of the dual-tower model, and the dual-tower model is trained based on the model input and the labels.

[0013] According to a third aspect of this disclosure, a text similarity calculation apparatus for a search scenario is provided, comprising:

[0014] The first acquisition module is used to acquire the first query text in the search request and acquire the first resource text of the resource to be processed; wherein the resource to be processed is the resource recalled based on the search request.

[0015] An add module is used to add prompt information to the first query text and the first resource text respectively, so as to obtain the second query text and the second resource text;

[0016] The extraction module is used to extract semantic information from the second query text and the second resource text respectively, to obtain a first semantic vector of the second query text and a second semantic vector of the second resource text;

[0017] The second acquisition module is used to calculate the similarity between the query text and the resource text based on the first semantic vector and the second semantic vector.

[0018] According to a fourth aspect of this disclosure, a dual-tower model training apparatus is provided, wherein the dual-tower model is used to compute text semantic similarity in a search scenario, the apparatus comprising:

[0019] The first acquisition module is used to acquire training data, which includes a first text sample pair and a label for the first text sample pair; wherein, the first text sample pair includes a first text and a second text.

[0020] An add module is used to add prompt information to the first text to obtain the third text, and to add prompt information to the second text to obtain the fourth text;

[0021] The training module is used to perform tokenization processing on the third text and the fourth text respectively to obtain the model input of the dual-tower model, and to train the dual-tower model based on the model input and the labels.

[0022] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0023] At least one processor; and

[0024] A memory that is communicatively connected to at least one processor; wherein,

[0025] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the text similarity calculation method for the search scenario described in the first aspect, or to perform the dual-tower model training method described in the second aspect.

[0026] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to execute the text similarity calculation method for a search scenario described in the first aspect above, or to execute the dual-tower model training method described in the second aspect above.

[0027] According to the seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the text similarity calculation method for a search scenario according to the first aspect above, or executes the dual-tower model training method of the second aspect above.

[0028] According to the text similarity calculation method for search scenarios disclosed herein, adding prompt information to the first query text and the first resource text can ensure that the input data structure of the dual-tower model and the large model is consistent. At the same time, based on the prompt information added to the first query text and the first resource text, the content of the first query text side and the first resource text side can learn from each other. This can solve the semantic loss problem caused by the difference in the model input data structure to a certain extent, improve model performance, and improve the accuracy of text similarity calculation.

[0029] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0030] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0031] Figure 1 This is a schematic diagram of a text similarity calculation method provided according to an embodiment of the present disclosure;

[0032] Figure 2 This is a flowchart illustrating a text similarity calculation method for a search scenario provided according to an embodiment of this disclosure;

[0033] Figure 3 This is a schematic diagram illustrating text similarity calculation in an advertising search scenario according to an embodiment of this disclosure;

[0034] Figure 4 This is a flowchart illustrating another text similarity calculation method for a search scenario provided according to an embodiment of this disclosure;

[0035] Figure 5 This is a flowchart illustrating a dual-tower model training method according to an embodiment of this disclosure;

[0036] Figure 6 This is a flowchart illustrating another dual-tower model training method provided according to an embodiment of the present disclosure;

[0037] Figure 7 This is a structural block diagram of a text similarity calculation device for a search scenario provided according to an embodiment of the present disclosure;

[0038] Figure 8 This is a structural block diagram of a dual-tower model training device provided according to an embodiment of the present disclosure;

[0039] Figure 9 This is a block diagram of an electronic device used to implement the text similarity calculation method or dual-tower model training method for search scenarios in the embodiments of this disclosure. Detailed Implementation

[0040] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0041] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0042] In related technologies, the problem of calculating similarity between texts (or semantic matching between massive amounts of data) can be solved using a "large model + extraction dual-tower" approach. For example... Figure 1 As shown, taking the advertising search scenario as an example, the "large model + dual-tower extraction" approach involves concatenating the query text (query) and candidate ad text (ad) as a whole and inputting them into the large model. Similarity is calculated using a semantic model and an MLP (Multi-Layer Perceptron). The large model is then used as a teacher to predict relevance soft labels from a massive dataset, resulting in training data for the dual-tower model. Finally, the pre-trained model from the large model is used as the initial model parameters, and model extraction techniques are employed to obtain an extraction dual-tower model with performance similar to the large model.

[0043] However, although the above-mentioned "large model + dual-tower extraction" approach enables deep interaction between query and ad content features under the large model, the input structure of the dual-tower model is still a single-sided query and a single-sided ad content rather than a whole concatenated query-ad content. The difference in the input data structure of the model before and after extraction leads to semantic loss and a significant loss in model accuracy.

[0044] Therefore, this disclosure provides a text similarity calculation method and a dual-tower model training method for search scenarios. Specifically, the text similarity calculation method and dual-tower model training method for search scenarios according to embodiments of this disclosure are described below with reference to the accompanying drawings.

[0045] Figure 2 This is a flowchart illustrating a text similarity calculation method for a search scenario provided according to an embodiment of this disclosure. It should be noted that the text similarity calculation method for a search scenario according to this disclosure can be applied to the text similarity calculation device for a search scenario according to an embodiment of this disclosure, which can be configured on an electronic device. For example... Figure 2 As shown, the text similarity calculation method for search scenarios may include the following steps:

[0046] Step 201: Obtain the first query text from the search request and the first resource text of the resource to be processed. The resource to be processed is the resource retrieved based on the search request.

[0047] As an example, when the search scenario is an advertising search scenario, the first query text can be the query text "query" in the user's search request, the resource to be processed is the candidate advertising resource recalled based on the search request, and the first resource text is the candidate advertising resource text "ad" of the resource to be processed.

[0048] Optionally, the search scenarios described in this disclosure can also be other search scenarios, such as news search scenarios, product name search scenarios, term search scenarios, etc., and this disclosure does not specifically limit them.

[0049] Step 202: Add prompt information to the first query text and the first resource text respectively to obtain the second query text and the second resource text.

[0050] In some embodiments of this disclosure, the prompt information includes N prompt slot virtual flag bits [MASK], where N is a positive integer.

[0051] Step 203: Extract semantic information from the second query text and the second resource text respectively to obtain the first semantic vector of the second query text and the second semantic vector of the second resource text.

[0052] As an example, semantic information can be extracted from the second query text and the second resource text by the semantic extraction module to obtain the first semantic vector of the second query text and the second semantic vector of the second resource text.

[0053] In one implementation, the second query text can be tokenized. During the tokenization process, a [CLS] flag is set at the beginning of each sentence in the second query text, and a [SEP] flag is set at the end of each sentence in both the first query text and the prompt message within the second query text, resulting in a tokenized result for the second query text. Similarly, the second resource text is tokenized, with a [CLS] flag set at the beginning of each sentence in the second resource text, and a [SEP] flag set at the end of each sentence in both the first resource text and the prompt message within the second resource text, resulting in a tokenized result for the second resource text. Then, semantic information is extracted from the second query text based on its tokenized result, yielding a first semantic vector for the second query text. Finally, semantic information is extracted from the second resource text based on its tokenized result, yielding a second semantic vector for the second resource text.

[0054] like Figure 3 As shown, taking an advertising search scenario as an example, two virtual flags [MASK] are added to the first query text to obtain the second query text. In other words, the text similarity calculation method for search scenarios in this embodiment of the present disclosure draws on the soft prompt concept and the mask mechanism in semantic models, at the dual-tower model input level. Construct virtual location for prompt slot While ensuring consistency in the input data structure of the large model and the extracted dual-tower model, a mask is used as the prompt content, enabling mutual prompting and learning between the query and ad sides. The second query text is tokenized, resulting in the tokenized result [CLS][MASK][MASK][SEP]query[SEP]. Two virtual flag bits [MASK] are added to the first resource text (ad) to obtain the second resource text. The second resource text is then tokenized, resulting in the tokenized result [CLS][MASK][MASK][SEP]ad[SEP].

[0055] It should be noted that in this example, the prompt message includes two virtual flag bits [MASK], which is only exemplary. In some embodiments of this disclosure, the number of [MASK] in the prompt message can be determined according to the actual situation.

[0056] Step 204: Calculate the similarity between the query text and the resource text based on the first semantic vector and the second semantic vector.

[0057] In one implementation, the cosine similarity between the first semantic vector and the second semantic vector can be calculated using cosine distance as the similarity between the query text and the resource text.

[0058] According to the text similarity calculation method for search scenarios in this disclosure, adding prompt information to the first query text and the first resource text can ensure that the input data structure of the dual-tower model and the large model is consistent. At the same time, based on the prompt information added to the first query text and the first resource text, the content of the first query text side and the first resource text side can learn from each other. This can solve the semantic loss problem caused by the difference in the model input data structure to a certain extent, improve model performance, and improve the accuracy of text similarity calculation.

[0059] Figure 4 This is a flowchart illustrating another text similarity calculation method for search scenarios provided according to embodiments of this disclosure. Figure 4 As shown, the text similarity calculation method for search scenarios may include the following steps:

[0060] Step 401: Obtain the first query text from the search request and the first resource text of the resource to be processed. The resource to be processed is the resource retrieved based on the search request.

[0061] Step 402: Add prompt information to the first query text and the first resource text respectively to obtain the second query text and the second resource text.

[0062] Step 403: Tokenize the second query text. During the tokenization process, set the [CLS] flag at the beginning of the second query text and set the [SEP] flag at the end of the first query text and the end of the prompt information in the second query text to obtain the tokenization result of the second query text.

[0063] Step 404: Tokenize the second resource text. During the tokenization process, set the [CLS] flag at the beginning of the second resource text and set the [SEP] flag at the end of the first resource text and the end of the prompt information in the second resource text to obtain the tokenization result of the second resource text.

[0064] Step 405: The tokenized result of the second query text is input into the first tower network of the pre-trained dual-tower model for semantic information extraction, resulting in the first semantic vector of the second query text. The dual-tower model is used to calculate semantic similarity.

[0065] In some embodiments of this disclosure, embedding calculations can be performed on the tokenized result of the second query text based on the embedding unit in the first pyramidal network to obtain a first text vector of the second query text. Semantic information is extracted from the second query text based on the semantic extraction unit in the first pyramidal network and the first text vector, and a first semantic vector of the second query text is generated based on the semantic vector of the prompt information in the second query text.

[0066] It should be noted that the training method for the dual-tower model in this embodiment can be found in the description in the subsequent embodiments of this disclosure, and will not be repeated here.

[0067] Step 406: Input the tokenized result of the second resource text into the second tower network in the dual-tower model to extract semantic information and obtain the second semantic vector of the second resource text.

[0068] In some embodiments of this disclosure, embedding calculations are performed on the tokenized result of the second resource text based on the embedding unit in the second pyramidal network to obtain a second text vector of the second resource text. Semantic information is extracted from the second resource text based on the semantic extraction unit in the second pyramidal network and the second text vector, and a second semantic vector of the second resource text is generated based on the semantic vector of the prompt information in the second resource text.

[0069] Step 407: Calculate the similarity between the query text and the resource text based on the first semantic vector and the second semantic vector.

[0070] In the embodiments of this disclosure, steps 401-404 and step 407 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit these steps and will not elaborate further.

[0071] According to the text similarity calculation method for search scenarios disclosed herein, prompt information is added to a first query text and a first resource text to obtain a second query text and a second resource text, and the second query text and the second resource text are respectively tokenized. Semantic information is extracted based on the tokenization results and a pre-trained dual-tower model, and then the similarity between the query text and the resource text is calculated. This disclosure can ensure that the input data structure of the dual-tower model is consistent with that of the large model, and, based on the prompt information added to the first query text and the first resource text, enable the content of the first query text and the first resource text to learn from each other, which can, to a certain extent, solve the semantic loss problem caused by differences in the model input data structure. Furthermore, this disclosure utilizes a pre-trained dual-tower model for calculating semantic similarity tasks to further improve the accuracy of text similarity calculation.

[0072] This disclosure also proposes a dual-tower model training method. This dual-tower model is used to compute text semantic similarity in search scenarios. Figure 5 This is a flowchart illustrating a dual-tower model training method according to an embodiment of this disclosure. It should be noted that the dual-tower model training method of this disclosure can be applied to the dual-tower model training device of this disclosure, which can be configured on an electronic device. Figure 5 As shown, the training method for this dual-tower model may include the following steps:

[0073] Step 501: Obtain training data, which includes a first text sample pair and its labels. The first text sample pair includes a first text and a second text.

[0074] It should be noted that the label of the first text sample pair can be a reference similarity between the first and second texts in the first text sample pair.

[0075] Step 502: Add a prompt message to the first text to obtain the third text, and add a prompt message to the second text to obtain the fourth text.

[0076] In some embodiments of this disclosure, the prompt information includes N prompt slot virtual flag bits [MASK], where N is a positive integer.

[0077] Step 503: The third and fourth texts are tokenized to obtain the model input of the dual-tower model, and the dual-tower model is trained based on the model input and labels.

[0078] According to the dual-tower model training method of this disclosure, adding prompt information to the first and second texts ensures that the input data structure of the dual-tower model is consistent with that of the larger model. Based on the prompt information, the content of the first and second texts learns from each other. The resulting dual-tower model can, to some extent, solve the semantic loss problem caused by differences in the model's input data structure, improve the performance of the dual-tower model in similarity calculation tasks, and increase the accuracy of text similarity calculation.

[0079] Figure 6 This is a flowchart illustrating another dual-tower model training method provided according to an embodiment of this disclosure. Figure 6 As shown, the training method for this dual-tower model may include the following steps:

[0080] Step 601: Obtain training data, which includes a first text sample pair and its labels. The first text sample pair includes a first text and a second text.

[0081] Step 602: Add a prompt message to the first text to obtain the third text, and add a prompt message to the second text to obtain the fourth text.

[0082] Step 603: Tokenize the third text. During the tokenization process, set the [CLS] flag at the beginning of the third text and set the [SEP] flag at the end of the first text and the end of the prompt message in the third text to obtain the tokenization result of the third text.

[0083] Step 604: Tokenize the fourth text. During the tokenization process, set the [CLS] flag at the beginning of the fourth text and set the [SEP] flag at the end of the second text and the end of the prompt message in the fourth text to obtain the tokenization result of the fourth text.

[0084] Step 605: Use the tokenization results of the third text and the fourth text as model inputs for the dual-tower model.

[0085] Step 606: Train the dual-tower model based on the model input and labels.

[0086] In one implementation, the tokenized result of the third text is input into the first pyramidal network of the dual-tower model to obtain the semantic vector of the prompt information in the third text output by the first pyramidal network. The tokenized result of the fourth text is input into the second pyramidal network of the dual-tower model to obtain the semantic vector of the prompt information in the fourth text output by the second pyramidal network. The semantic vectors of the prompt information in the third and fourth texts are then input into the semantic similarity calculation unit of the dual-tower model to obtain the similarity between the first and second texts. Based on the similarity and the label, a loss value for the dual-tower model is generated, and the dual-tower model is trained based on the loss value.

[0087] In the embodiments of this disclosure, steps 601 and 602 can be implemented in any of the various embodiments of this disclosure. This disclosure does not limit these steps and will not elaborate further.

[0088] According to the dual-tower model training method of this disclosure, adding prompt information to the first and second texts ensures that the input data structure of the dual-tower model is consistent with that of the larger model. Based on the prompt information, the content of the first and second texts learns from each other through mutual prompting. The third and fourth texts, to which the prompt information has been added, are then tokenized, and the tokenization results are used as the model input for the dual-tower model. This training method obtains the dual-tower model and can, to some extent, solve the semantic loss problem caused by differences in the model's input data structure, further improving the performance of the dual-tower model in similarity calculation tasks and increasing the accuracy of text similarity calculation.

[0089] Figure 7 This is a structural block diagram of a text similarity calculation device for a search scenario provided according to an embodiment of this disclosure. Figure 7 As shown, the text similarity calculation device for search scenarios includes: a first acquisition module 701, an addition module 702, an extraction module 703, and a second acquisition module 704.

[0090] in,

[0091] The first acquisition module 701 is used to acquire the first query text in the search request and acquire the first resource text of the resource to be processed; wherein the resource to be processed is the resource recalled based on the search request.

[0092] Add module 702, which is used to add prompt information to the first query text and the first resource text respectively, to obtain the second query text and the second resource text.

[0093] The extraction module 703 is used to extract semantic information from the second query text and the second resource text respectively, so as to obtain the first semantic vector of the second query text and the second semantic vector of the second resource text.

[0094] The second acquisition module 704 is used to calculate the similarity between the query text and the resource text based on the first semantic vector and the second semantic vector.

[0095] In some embodiments of this disclosure, the extraction module 703 is specifically used for: tokenizing the second query text, and during the tokenization process, setting a [CLS] flag at the beginning of the second query text, and setting a [SEP] flag at the end of the first query text and the end of the prompt information in the second query text, to obtain the tokenization result of the second query text; tokenizing the second resource text, and during the tokenization process, setting a [CLS] flag at the beginning of the second resource text, and setting a [SEP] flag at the end of the first resource text and the end of the prompt information in the second resource text, to obtain the tokenization result of the second resource text; extracting semantic information from the second query text based on the tokenization result of the second query text to obtain a first semantic vector of the second query text, and extracting semantic information from the second resource text based on the tokenization result of the second resource text to obtain a second semantic vector of the second resource text.

[0096] In some embodiments of this disclosure, the extraction module 703 is specifically used to: input the tokenized result of the second query text into the first tower network in the pre-trained dual-tower model to extract semantic information and obtain a first semantic vector of the second query text; wherein, the dual-tower model is used to calculate the semantic similarity task; input the tokenized result of the second resource text into the second tower network in the dual-tower model to extract semantic information and obtain a second semantic vector of the second resource text.

[0097] In some embodiments of this disclosure, the extraction module 703 is specifically used to: perform embedding calculation on the tokenization result of the second query text based on the embedding unit in the first pyramidal network to obtain a first text vector of the second query text; extract semantic information from the second query text based on the semantic extraction unit in the first pyramidal network and the first text vector, and generate a first semantic vector of the second query text based on the semantic vector of the prompt information in the second query text.

[0098] In some embodiments of this disclosure, the extraction module 703 is specifically used to: perform embedding calculation on the tokenization result of the second resource text based on the embedding unit in the second pyramidal network to obtain a second text vector of the second resource text; extract semantic information from the second resource text based on the semantic extraction unit in the second pyramidal network and the second text vector, and generate a second semantic vector of the second resource text according to the semantic vector of the prompt information in the second resource text.

[0099] In some embodiments of this disclosure, the prompt information includes N prompt slot virtual flag bits [MASK], where N is a positive integer.

[0100] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0101] Figure 8 This is a structural block diagram of a dual-tower model training device provided according to an embodiment of the present disclosure.

[0102] This dual-tower model is used to calculate text semantic similarity in search scenarios. For example... Figure 8 As shown, the dual-tower model training device includes: a first acquisition module 801, an addition module 802, and a training module 803. Wherein,

[0103] The first acquisition module 801 is used to acquire training data, the training data including a first text sample pair and a label for the first text sample pair; wherein, the first text sample pair includes a first text and a second text.

[0104] Adding module 802 is used to add prompt information to the first text to obtain the third text, and to add prompt information to the second text to obtain the fourth text.

[0105] The training module 803 is used to perform tokenization processing on the third text and the fourth text respectively to obtain the model input of the dual-tower model, and to train the dual-tower model based on the model input and the labels.

[0106] In some embodiments of this disclosure, the training module 803 is specifically used to: tokenize the third text, and during the tokenization process, set a [CLS] flag at the beginning of the third text, and set a [SEP] flag at the end of the first text and the end of the prompt information in the third text, to obtain the tokenization result of the third text; tokenize the fourth text, and during the tokenization process, set a [CLS] flag at the beginning of the fourth text, and set a [SEP] flag at the end of the second text and the end of the prompt information in the fourth text, to obtain the tokenization result of the fourth text; and use the tokenization results of the third text and the fourth text as model inputs to the dual-tower model.

[0107] In some embodiments of this disclosure, the training module 803 is specifically used for: inputting the tokenized result of the third text into the first pyramidal network of the dual-tower model to obtain the semantic vector of the prompt information in the third text output by the first pyramidal network; inputting the tokenized result of the fourth text into the second pyramidal network of the dual-tower model to obtain the semantic vector of the prompt information in the fourth text output by the second pyramidal network; inputting the semantic vector of the prompt information in the third text and the semantic vector of the prompt information in the fourth text into the semantic similarity calculation unit of the dual-tower model to obtain the similarity between the first text and the second text; generating the loss value of the dual-tower model based on the similarity and the label, and training the dual-tower model based on the loss value.

[0108] In some embodiments of this disclosure, the prompt information includes N prompt slot virtual flag bits [MASK], where N is a positive integer.

[0109] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0110] According to embodiments of the present disclosure, the present disclosure also provides an electronic device and a computer-readable storage medium.

[0111] like Figure 9 As shown, Figure 9 This is a block diagram of an electronic device used to implement the text similarity calculation method or dual-tower model training method for search scenarios according to embodiments of this disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the disclosure described and / or claimed herein.

[0112] like Figure 9 As shown, the electronic device includes one or more processors 901, a memory 902, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 9 Take the 901 processor as an example.

[0113] The memory 902 is the non-transitory computer-readable storage medium provided in this disclosure. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the text similarity calculation method or the dual-tower model training method for search scenarios provided in this disclosure. The non-transitory computer-readable storage medium of this disclosure stores computer instructions for causing a computer to perform the text similarity calculation method or the dual-tower model training method for search scenarios provided in this disclosure.

[0114] Memory 902, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the text similarity calculation method or the dual-tower model training method for the search scenario in the embodiments of this disclosure (e.g., attached). Figure 7 The first acquisition module 701, the addition module 702, the extraction module 703, and the second acquisition module 704 shown, along with the attached... Figure 8The first acquisition module 801, the addition module 802, and the training module 803 are shown. The processor 901 executes various server functions and data processing by running non-transient software programs, instructions, and modules stored in the memory 902, that is, implementing the text similarity calculation method or the dual-tower model training method for search scenarios in the above method embodiments.

[0115] The memory 902 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function. The data storage area may store data created by the use of the electronic device for text similarity calculation methods or dual-tower model training methods in a search scenario. Furthermore, the memory 902 may include high-speed random access memory and may also include non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 902 may optionally include memory remotely located relative to the processor 901. This remote memory can be connected via a network to the electronic device used to implement the text similarity calculation method or dual-tower model training method for a search scenario. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0116] The electronic device used for text similarity calculation methods or dual-tower model training methods in search scenarios may further include: an input device 903 and an output device 904. The processor 901, memory 902, input device 903, and output device 904 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0117] Input device 903 can receive input numerical or character information, as well as key signal input related to user settings and function control of electronic devices used for text similarity calculation methods or dual-tower model training methods in search scenarios. Examples of input devices include touchscreens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, and joysticks. Output device 904 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touchscreen.

[0118] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs, i.e., this disclosure also proposes a computer program that, when executed by a processor, implements the text similarity calculation method or dual-tower model training method for searching a scenario described in the above embodiments. This one or more computer programs can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0121] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0122] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server," or simply "VPS") in terms of management difficulty and weak business scalability. The server can also be a server for a distributed system or a server integrated with blockchain. It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure is achieved, and this is not limited herein.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A text similarity calculation method for search scenarios, comprising: Obtain the first query text from the search request and obtain the first resource text of the resource to be processed; wherein the resource to be processed is the resource recalled based on the search request; Add prompt information to the first query text and the first resource text respectively to obtain the second query text and the second resource text; the prompt information includes N prompt slot virtual flag bits [MASK], where N is a positive integer; Semantic information extraction is performed on the second query text and the second resource text respectively to obtain a first semantic vector of the second query text and a second semantic vector of the second resource text. This includes: tokenizing the second query text and the second resource text respectively to obtain tokenized results for the second query text and the second resource text; inputting the tokenized result of the second query text into a first pyramidal network of a pre-trained dual-pyramid model for semantic information extraction to obtain the first semantic vector of the second query text; and inputting the tokenized result of the second resource text into a second pyramidal network of the dual-pyramid model for semantic information extraction to obtain the second semantic vector of the second resource text; wherein the dual-pyramid model is used to calculate semantic similarity. Calculate the similarity between the query text and the resource text based on the first semantic vector and the second semantic vector.

2. The method as described in claim 1, wherein, The step of tokenizing the second query text and the second resource text respectively to obtain the tokenized result of the second query text and the tokenized result of the second resource text includes: The second query text is tokenized, and during the tokenization process, a [CLS] flag is set at the beginning of the second query text, and a [SEP] flag is set at the end of the first query text and the end of the prompt information in the second query text, so as to obtain the tokenization result of the second query text; The second resource text is tokenized, and during the tokenization process, a [CLS] flag is set at the beginning of the second resource text, and a [SEP] flag is set at the end of the first resource text and the end of the prompt information in the second resource text, so as to obtain the tokenization result of the second resource text.

3. The method as described in claim 1, wherein, The step of inputting the tokenized result of the second query text into the first tower network of the pre-trained dual-tower model for semantic information extraction, to obtain the first semantic vector of the second query text, includes: Embedding calculations are performed on the tokenized result of the second query text based on the embedding unit in the first pyramidal network to obtain the first text vector of the second query text; Based on the semantic extraction unit in the first pyramidal network and the first text vector, semantic information is extracted from the second query text, and a first semantic vector of the second query text is generated according to the semantic vector of the prompt information in the second query text.

4. The method of claim 1, wherein, The step of inputting the tokenized result of the second resource text into the second tower network of the dual-tower model for semantic information extraction, to obtain the second semantic vector of the second resource text, includes: Embedding calculations are performed on the tokenized result of the second resource text based on the embedding units in the second pyramidal network to obtain the second text vector of the second resource text; The semantic information of the second resource text is extracted based on the semantic extraction unit in the second pyramidal network and the second text vector, and a second semantic vector of the second resource text is generated based on the semantic vector of the prompt information in the second resource text.

5. A dual-tower model training method, wherein, The dual-tower model is used to calculate text semantic similarity in a search scenario, and the method includes: Acquire training data, which includes a first text sample pair and a label for the first text sample pair; wherein the first text sample pair includes a first text and a second text; Add a prompt message to the first text to obtain the third text, and add a prompt message to the second text to obtain the fourth text; the prompt message includes N prompt slot virtual flag bits [MASK], where N is a positive integer; The third text and the fourth text are respectively tokenized to obtain the model input of the dual-tower model, including: tokenizing the third text, and during the tokenization process, setting a [CLS] flag at the beginning of the third text and a [SEP] flag at the end of the first text and the prompt information within the third text, to obtain the tokenization result of the third text; tokenizing the fourth text, and during the tokenization process, setting a [CLS] flag at the beginning of the fourth text and a [SEP] flag at the end of the second text and the prompt information within the fourth text, to obtain the tokenization result of the fourth text; using the tokenization results of the third text and the fourth text as the model input of the dual-tower model; Training the dual-tower model based on the model input and the labels includes: inputting the tokenized result of the third text into the first tower network of the dual-tower model to obtain the semantic vector of the prompt information in the third text output by the first tower network; inputting the tokenized result of the fourth text into the second tower network of the dual-tower model to obtain the semantic vector of the prompt information in the fourth text output by the second tower network; inputting the semantic vectors of the prompt information in the third text and the fourth text into the semantic similarity calculation unit of the dual-tower model to obtain the similarity between the first text and the second text; generating the loss value of the dual-tower model based on the similarity and the labels, and training the dual-tower model based on the loss value.

6. A text similarity calculation device for a search scenario, comprising: The first acquisition module is used to acquire the first query text in the search request and acquire the first resource text of the resource to be processed; wherein the resource to be processed is the resource recalled based on the search request. An adding module is used to add prompt information to the first query text and the first resource text respectively, to obtain the second query text and the second resource text; the prompt information includes N prompt slot virtual flag bits [MASK], where N is a positive integer; An extraction module is used to extract semantic information from the second query text and the second resource text respectively, to obtain a first semantic vector of the second query text and a second semantic vector of the second resource text. This includes: tokenizing the second query text and the second resource text respectively, obtaining tokenized results for the second query text and the second resource text; inputting the tokenized result of the second query text into a first pyramidal network in a pre-trained dual-pyramid model for semantic information extraction, to obtain the first semantic vector of the second query text; and inputting the tokenized result of the second resource text into a second pyramidal network in the dual-pyramid model for semantic information extraction, to obtain the second semantic vector of the second resource text; wherein, the dual-pyramid model is used to calculate semantic similarity. The second acquisition module is used to calculate the similarity between the query text and the resource text based on the first semantic vector and the second semantic vector.

7. The apparatus of claim 6, wherein, The extraction module is specifically used for: The second query text is tokenized, and during the tokenization process, a [CLS] flag is set at the beginning of the second query text, and a [SEP] flag is set at the end of the first query text and the end of the prompt information in the second query text, so as to obtain the tokenization result of the second query text; The second resource text is tokenized, and during the tokenization process, a [CLS] flag is set at the beginning of the second resource text, and a [SEP] flag is set at the end of the first resource text and the end of the prompt information in the second resource text, so as to obtain the tokenization result of the second resource text.

8. The apparatus of claim 6, wherein, The extraction module is specifically used for: Embedding calculations are performed on the tokenized result of the second query text based on the embedding unit in the first pyramidal network to obtain the first text vector of the second query text; Based on the semantic extraction unit in the first pyramidal network and the first text vector, semantic information is extracted from the second query text, and a first semantic vector of the second query text is generated according to the semantic vector of the prompt information in the second query text.

9. The apparatus of claim 6, wherein, The extraction module is specifically used for: Embedding calculations are performed on the tokenized result of the second resource text based on the embedding units in the second pyramidal network to obtain the second text vector of the second resource text; The semantic information of the second resource text is extracted based on the semantic extraction unit in the second pyramidal network and the second text vector, and a second semantic vector of the second resource text is generated based on the semantic vector of the prompt information in the second resource text.

10. A dual-tower model training device, wherein, The dual-tower model is used to calculate text semantic similarity in a search scenario, and the device includes: The first acquisition module is used to acquire training data, which includes a first text sample pair and a label for the first text sample pair; wherein, the first text sample pair includes a first text and a second text. An adding module is used to add prompt information to the first text to obtain the third text, and to add prompt information to the second text to obtain the fourth text; the prompt information includes N prompt slot virtual flag bits [MASK], where N is a positive integer; The training module is used to perform tokenization processing on the third text and the fourth text respectively to obtain the model input of the dual-tower model, and to train the dual-tower model based on the model input and the labels; Specifically, the training module is used to: tokenize the third text, and during the tokenization process, set a [CLS] flag at the beginning of the third text, and set a [SEP] flag at the end of the first text and the end of the prompt information within the third text, to obtain the tokenization result of the third text; tokenize the fourth text, and during the tokenization process, set a [CLS] flag at the beginning of the fourth text, and set a [SEP] flag at the end of the second text and the end of the prompt information within the fourth text, to obtain the tokenization result of the fourth text; and use the tokenization results of the third text and the fourth text as model inputs to the dual-tower model; The training module is specifically used for: inputting the tokenized result of the third text into the first pyramidal network of the dual-tower model to obtain the semantic vector of the prompt information in the third text output by the first pyramidal network; inputting the tokenized result of the fourth text into the second pyramidal network of the dual-tower model to obtain the semantic vector of the prompt information in the fourth text output by the second pyramidal network; inputting the semantic vector of the prompt information in the third text and the semantic vector of the prompt information in the fourth text into the semantic similarity calculation unit of the dual-tower model to obtain the similarity between the first text and the second text; generating the loss value of the dual-tower model based on the similarity and the label, and training the dual-tower model based on the loss value.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 4, or to perform the method of claim 5.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1 to 4, or to cause the computer to perform the method as described in claim 5.

13. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method of any one of claims 1 to 4, or the steps of the method of claim 5.

Images