Semantic vector extraction model training and semantic vector representation method and device

Abstract

The present application discloses a method and apparatus for training a semantic vector extraction model and representing semantic vectors. The method for training the semantic vector extraction model includes: obtaining a conversation corpus from the same scenario as a downstream task corresponding to the semantic vector as a first pre-training text; performing secondary training on the trained BERT language network based on the first pre-training text to generate a BERT language network that focuses on nouns and verbs in the text; obtaining text classification data and generating a text similarity dataset based on the text classification data; and fine-tuning a multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for training a semantic vector extraction model and representing a semantic vector. Background Art

[0002] Semantic recognition is widely used in search and dialogue systems. For example, when a customer enters a query into an intelligent dialogue system, the backend will identify the semantic information in the text and return an answer. The underlying algorithm logic is generally as follows: first, business data and user log data are collected to maintain a FAQ database in the form of "question-answer" pairs. Then, the standard questions in the question-answer pairs are expanded to generate multiple similar questions for semantic matching. When a query is entered, the system first performs a rough recall, searching the FAQ database for the questionlist with the highest degree of match to the query text. A larger model with higher accuracy is then used for fine sorting. If the accuracy exceeds a set threshold, the relevant answer is returned to the customer.

[0003] There are two common approaches to semantic representation: 1. Using traditional machine learning algorithms such as BM25 and TF-IDF for literal text representation is fast but lacks semantic and word order information, making it difficult to handle semantic diversity, synonyms, and polysemy in queries, resulting in low-quality semantic representation. 2. Using deep learning models such as LSTM and BERT with Siamese representational architectures for text similarity training, these models are equipped with semantic extraction capabilities. While this method can pre-calculate semantic vectors and accelerate semantic recall, it lacks internal text interaction and has limitations in semantic representation.

[0004] The above-mentioned existing methods for semantically representing text data have technical problems such as low semantic representation quality and limited semantic expression, and no effective solution has been proposed yet. Summary of the Invention

[0005] The embodiments of the present application provide a training method, device and storage medium for a semantic vector extraction model to at least solve the technical problems in the prior art of semantically representing text data, such as low semantic representation quality and low semantic expression limitations.

[0006] According to one aspect of an embodiment of the present application, a training method for a semantic vector extraction model is provided, comprising: obtaining a dialogue corpus in the same scenario as a downstream task corresponding to a semantic vector as a first pre-training text, and performing secondary training on a trained BERT language network based on the first pre-training text to generate a BERT language network that focuses on nouns and verbs in the text; obtaining text classification data, and generating a text similarity dataset based on the text classification data, wherein the text classification data is classified text data, and the text similarity dataset consists of the text classification data and other text data with different degrees of similarity to the text classification data; and fine-tuning a multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model, wherein the multi-task classification network includes two BERT language networks after secondary training, two attention mechanism networks, two multi-classification fully connected layers, and one semantic matching layer, and the semantic vector extraction model includes the BERT language network and the attention mechanism network in the multi-task classification network generated at the end of the fine-tuning training.

[0007] According to another aspect of an embodiment of the present application, a semantic vector representation method is also provided, which is applicable to the semantic vector extraction model trained by the above method, including: obtaining the text to be recognized; and inputting the text to be recognized into the semantic vector extraction model to obtain the semantic vector of the text to be recognized.

[0008] According to another aspect of an embodiment of the present application, a storage medium is further provided, the storage medium including a stored program, wherein when the program is running, a processor executes any one of the above methods.

[0009] According to another aspect of an embodiment of the present application, a training device for a semantic vector extraction model is also provided, including: a first generation module, used to obtain dialogue corpus of the same scenario as the downstream task corresponding to the semantic vector as a first pre-training text, and based on the first pre-training text, perform secondary training on the trained BERT language network to generate a BERT language network that focuses on nouns and verbs in the text; a second generation module, used to obtain text classification data, and generate a text similarity dataset based on the text classification data, wherein the text classification data is classified text data, and the text similarity dataset consists of text classification data and other text data with different degrees of similarity to the text classification data; and a third generation module, used to fine-tune the multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model, wherein the multi-task classification network includes two BERT language networks after secondary training, two attention mechanism networks, two multi-classification fully connected layers and one semantic matching layer, and the semantic vector extraction model includes the BERT language network and attention mechanism network in the multi-task classification network generated at the end of fine-tuning training.

[0010] According to another aspect of an embodiment of the present application, a semantic vector representation device is also provided, which is suitable for a semantic vector extraction model trained by the above-mentioned device, including: a first acquisition module for acquiring text to be recognized; and a second acquisition module for inputting the text to be recognized into the semantic vector extraction model to obtain the semantic vector of the text to be recognized.

[0011] According to another aspect of an embodiment of the present application, a training device for a semantic vector extraction model is also provided, comprising: a first processor; and a first memory connected to the first processor, for providing the first processor with instructions for processing the following processing steps: obtaining a dialogue corpus of the same scene as the downstream task corresponding to the semantic vector as a first pre-training text, and performing secondary training on the trained BERT language network based on the first pre-training text to generate a BERT language network that focuses on nouns and verbs in the text; obtaining text classification data, and generating a text similarity dataset based on the text classification data, wherein the text classification data is classified text data, and the text similarity dataset consists of the text classification data and other text data with different degrees of similarity to the text classification data; and fine-tuning the multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model, wherein the multi-task classification network includes two BERT language networks after secondary training, two attention mechanism networks, two multi-classification fully connected layers and one semantic matching layer, and the semantic vector extraction model includes the BERT language network and the attention mechanism network in the multi-task classification network generated at the end of fine-tuning training.

[0012] According to another aspect of an embodiment of the present application, a semantic vector representation device is also provided, which is suitable for a semantic vector extraction model trained by the above-mentioned device, including: a second processor; and a second memory connected to the second processor, for providing instructions for the second processor to process the following processing steps: obtaining text to be recognized; and inputting the text to be recognized into the semantic vector extraction model to obtain a semantic vector of the text to be recognized.

[0013] In an embodiment of the present application, a computing device uses a conversation corpus of the same scene as the downstream task corresponding to the semantic vector as the first pre-training text, and performs secondary training on the trained BERT language network based on the first pre-training text to generate a BERT language network that focuses on nouns and verbs in the text, thereby making the BERT language network pay more attention to nouns and verbs with higher weights in text matching. Afterwards, the computing device obtains text classification data, generates a text similarity dataset based on the text classification data, and fine-tunes the multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model. Thus, the present technical solution utilizes a multi-task classification network to fuse text classification and text similarity models. Feature learning is performed on the global representation and local comparison of the text, thereby improving the semantic understanding ability of the multi-task classification network.

[0014] Therefore, this technical solution uses dialogue materials to further pre-train the BERT language network, and the training task is changed to predicting words of specific parts of speech, so that the BERT language network has a better understanding of nouns and verbs with higher weights in similarity calculations. Compared with existing learning algorithms, this technical solution is based on the BERT language network and introduces multi-task training that integrates model similarity and text classification. By learning the connections and differences between different tasks, the quality of semantic representation is improved, and problems such as missing semantics and word order information will not occur. The internal interaction of the text makes the semantic expression unlimited, and also reduces the risk of model overfitting. The attention mechanism is introduced as a weak interaction in the representational semantic matching model to reduce the interference of redundant text, thereby improving the accuracy of the model. The technical effect of high quality of semantic representation and low limitation of semantic expression is achieved, thereby solving the technical problem of low quality of semantic representation and low limitation of semantic expression in the existing technology for semantic representation of text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to embodiment 1 of the present application;

[0017] Figure 2 1 is a flow chart of a method for training a semantic vector extraction model according to the first aspect of Example 1 of the present application;

[0018] Figure 3 1 is a schematic diagram of the process of BERT language network pre-training according to the first aspect of Example 1 of the present application;

[0019] Figure 4 2 is a flow chart of fine-tuning training of a multi-task classification network according to the first aspect of Example 1 of the present application;

[0020] Figure 5 1 is a flowchart of the semantic vector extraction model reasoning process according to the first aspect of Example 1 of the present application;

[0021] Figure 6 1 is a flow chart of the semantic vector representation method according to the second aspect of Example 1 of the present application;

[0022] Figure 7 2 is a schematic diagram of a training device for a semantic vector extraction model according to the first aspect of Example 2 of the present application;

[0023] Figure 8 is a schematic diagram of a semantic vector representation device according to the second aspect of Example 2 of the present application;

[0024] Figure 9 is a schematic diagram of a training device for a semantic vector extraction model according to the first aspect of Example 3 of the present application; and

[0025] Figure 10 It is a schematic diagram of the semantic vector representation device according to the second aspect of Example 3 of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Example 1

[0029] According to this embodiment, a training method for a semantic vector extraction model and an embodiment of a semantic vector representation method are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiment provided in this embodiment can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Figure 1 The hardware structure block diagram of a computing device for implementing a training method for a semantic vector extraction model and a semantic vector representation method is shown. Figure 1 As shown, the computing device may include one or more processors (the processor may include but is not limited to a microprocessor MCU or a programmable logic device FPGA, etc.), a memory for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0031] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computing device. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the training method of the semantic vector extraction model and the semantic vector representation method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the training method of the semantic vector extraction model and the semantic vector representation method of the above-mentioned application. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The transmission device is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communications provider of the computing device. In one embodiment, the transmission device includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computing device.

[0035] It should be noted that, in some optional embodiments, the above Figure 1 The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing device described above.

[0036] In the above operating environment, according to the first aspect of this embodiment, a training method for a semantic vector extraction model is provided. Figure 2 A schematic diagram of the process is shown in FIG. Figure 2 As shown, the method includes:

[0037] S202: Acquire a conversation corpus corresponding to the semantic vector and belonging to the same scenario as the downstream task as a first pre-training text, and perform a second training on the trained BERT language network based on the first pre-training text to generate a BERT language network that focuses on nouns and verbs in the text;

[0038] S204: Acquire text classification data, and generate a text similarity dataset based on the text classification data, wherein the text classification data is classified text data, and the text similarity dataset consists of the text classification data and other text data having different degrees of similarity with the text classification data; and

[0039] S206: Based on the text similarity dataset, fine-tune the multi-task classification network to generate a semantic vector extraction model, wherein the multi-task classification network includes two BERT language networks after secondary training, two attention mechanism networks, two multi-classification fully connected layers and one semantic matching layer, and the semantic vector extraction model includes the BERT language network and attention mechanism network in the multi-task classification network generated at the end of fine-tuning training.

[0040] Specifically, the computing device obtains the conversation corpus of the same scene as the downstream task corresponding to the semantic vector as the first pre-training text. For example, the downstream task corresponding to the semantic vector may be to calculate semantic similarity, and the computing device obtains the conversation corpus of the same scene as the task of calculating semantic similarity as the first pre-training text. Then, based on the first pre-training text, the computing device performs secondary training on the trained BERT language network to generate a BERT language network that focuses on nouns and verbs in the text. Specifically, refer to Figure 3 As shown, the MLM task in the trained BERT language network uses a "random word mask" to predict masked words. Based on the first pre-trained text, the computing device replaces the "random word mask" in the MLM task with a "word mask of a specific part of speech" for the trained BERT language network and performs a second training. The specific parts of speech are nouns and verbs. The computing device then generates a BERT language network that focuses on the nouns and verbs in the text.

[0041] Furthermore, the computing device obtains text classification data and generates a text similarity dataset based on the text classification data. The text classification data is classified text data, and the text similarity dataset is composed of the text classification data and other text data with different degrees of similarity to the text classification data. For example, the text classification data is "Text: How to apply for a cash loan, label: How to apply", and the computing device generates a text similarity dataset based on the text classification data, such as "Text1: How to apply for a cash loan, Text2: How to apply for a loan, label: Similar", "Text1: How to apply for a cash loan, Text2: How to apply for an online loan, label: Similar", "Text1: How to apply for a cash loan, Text2: How to apply for a salary increase, label: Dissimilar", and "Text1: How to apply for a cash loan, Text2: How to apply for a promotion, label: Dissimilar", etc.

[0042] Furthermore, the computing device fine-tunes the multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model. Specifically, as shown in FIG4 , the computing device fine-tunes the multi-task classification network comprising two BERT language networks, two attention mechanism networks (Attention Layer), two multi-classification fully connected layers (Dense Layer) and a semantic matching layer (Sim Layer) after secondary training based on the text similarity dataset. When the data output by a task layer in the multi-task classification network is inconsistent with the pre-labeled data, the task layer in the multi-task classification network is fine-tuned. Moreover, after the fine-tuning training of the multi-task classification network is completed, the BERT language network and attention mechanism network in the multi-task classification network can be used when calculating the semantic vector of a certain text data offline, thereby outputting the corresponding semantic vector. Therefore, the computing device can combine the BERT language network and attention mechanism network (Attention Layer) in the multi-task classification network generated at the end of the fine-tuning training to form a semantic vector extraction model specifically for semantic vector representation of text data.

[0043] As described in the background technology, there are two conventional methods for semantic representation: 1. Using traditional machine learning algorithms such as BM25 and TF-IDF to represent text literally. This is fast but lacks semantic and word order information, making it difficult to handle semantic expression diversity, synonyms, and polysemy in query problems, resulting in low-quality semantic representation. 2. Using deep learning models such as LSTM and BERT with Siamese representational structures to train text similarity, giving the model semantic extraction capabilities. Although this method can calculate semantic vectors in advance and speed up semantic recall, it lacks internal interaction within the text and has limitations in semantic expression.

[0044] In response to the technical problems described above, the computing device uses the dialogue corpus of the same scenario as the downstream task corresponding to the semantic vector as the first pre-training text, and performs secondary training on the trained BERT language network based on the first pre-training text to generate a BERT language network that focuses on nouns and verbs in the text, thereby making the BERT language network pay more attention to nouns and verbs with higher weights in text matching. Afterwards, the computing device obtains text classification data, generates a text similarity dataset based on the text classification data, and fine-tunes the multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model. Thus, this technical solution utilizes a multi-task classification network to fuse text classification and text similarity models. Feature learning is performed on the global representation and local comparison of the text, thereby improving the semantic understanding ability of the multi-task classification network.

[0045] Therefore, this technical solution uses dialogue materials to further pre-train the BERT language network, and the training task is changed to predicting words of specific parts of speech, so that the BERT language network has a better understanding of nouns and verbs with higher weights in similarity calculations. Compared with existing learning algorithms, this technical solution is based on the BERT language network and introduces multi-task training that integrates model similarity and text classification. By learning the connections and differences between different tasks, the quality of semantic representation is improved, and problems such as missing semantics and word order information will not occur. The internal interaction of the text makes the semantic expression unlimited, and also reduces the risk of model overfitting. The attention mechanism is introduced as a weak interaction in the representational semantic matching model to reduce the interference of redundant text, thereby improving the accuracy of the model. The technical effect of high quality of semantic representation and low limitation of semantic expression is achieved, thereby solving the technical problem of low quality of semantic representation and low limitation of semantic expression in the existing technology for semantic representation of text data.

[0046] Optionally, based on the first pre-trained text, the trained BERT language network is retrained to generate a BERT language network that focuses on nouns and verbs in the text, including: processing the first pre-trained text to generate an input text in a single-round dialogue format; performing word segmentation tagging and part-of-speech tagging on the input text, wherein the part-of-speech tagging of the input text includes noun tagging and verb tagging on the input text; based on the results of the word segmentation tagging and part-of-speech tagging, using a mask to replace a predetermined number of nouns and verbs in the input text; and based on the masked input text, the trained BERT language network is retrained to enable the BERT language network to predict the masked words, thereby generating a BERT language network that focuses on the nouns and verbs in the text.

[0047] Specifically, the computing device processes the first pre-trained text to generate an input text in a single-round dialogue format. Figure 3As shown, the first training text is "Hello, can you help me check the remaining debt?". The single-round dialogue format is "[cls][token1 token2 token3…][sep]". Here, token is used to identify the word segments in the text. [cls] means the output at the position of the special character, representing the semantic vector of the entire text. [sep] means the special character indicating the end of the sentence. In this text, a position vector is also superimposed to represent the relative position of the text. The computing device generates the input text in the single-round dialogue format for this first training text as "clsHello, can you help me check the remaining debtsep". The computing device performs word segmentation tagging and词性 tagging on this input text. The词性 tagging of the input text includes noun tagging and verb tagging of the input text. Then, based on the results of the word segmentation tagging and词性 tagging, the computing device uses masks to replace a predetermined number of nouns and verbs in the input text. For example, after the computing device performs word segmentation tagging on this input text, the word segments obtained are: "Hello", "can you", "help me", "check", "the", "remaining", and "debt". After the computing device performs词性 tagging on this input text, it tags "check" in the input text as a verb and "debt" as a noun. Then, the computing device uses the mask Mask to replace the noun "debt" and the verb "check" in the input text. Then, based on the input text "clsHello, can you help me Mask the remaining Masksep" after mask processing, the computing device performs secondary training on the pre-trained BERT language network so that the BERT language network predicts the masked words, thereby generating a BERT language network that focuses on nouns and verbs in the text.

[0048] Therefore, this technical solution performs secondary pre-training on the BERT language network based on the downstream task of the same-scenario dialogue corpus, changing the "random word Mask" in the MLM task during pre-training to "words of specific词性 Mask", so that the BERT language network pays more attention to entity nouns and verbs with higher weights in text matching, and thus can more accurately mine semantic information. Compared with the prior art, this technical solution further pre-trains the BERT language network using the dialogue corpus, and changes the training task to predicting words of specific词性, enabling the BERT language network to have a better understanding of nouns and verbs with higher weights in similarity calculation.

[0049] It should be noted that the term "词性" in the original text seems a bit unclear. It might be more appropriate to use a more specific term like "grammatical category" or something more contextually relevant. But I've translated it as "词性" as per the original text's requirement.Optionally, the other text data having different degrees of similarity with the text classification data include similar text data and dissimilar text data, and the operation of generating a text similarity dataset based on the text classification data includes: determining similar text data of the same type as the text classification data from a preset text classification dataset, and randomly constructing similar text data based on the similar text data, wherein the preset text classification dataset is composed of text classification data of different categories; determining non-similar text data of a different type from the text classification data from the preset text classification dataset, and constructing dissimilar text data based on the non-similar text data; and constructing a text similarity dataset based on the similar text data and the dissimilar text data, wherein the ratio of similar text data to dissimilar text data included in the text similarity dataset is 1:5.

[0050] Specifically, other text data with varying degrees of similarity to the text classification data includes similar text data and dissimilar text data. The computing device determines similar text data of the same type as the text classification data from a preset text classification dataset, and randomly constructs similar text data based on the similar text data. The preset text classification dataset is composed of text classification data of different categories. For example, the text classification data "How to apply for a cash loan" is of the type "how to apply." The computing device determines similar text data of the same type as the text classification data "How to apply for a cash loan" from the preset text classification dataset, namely, similar text data of the type "how to apply." The computing device randomly selects a certain number of similar text data from the similar text data to construct similar text data. The computing device then determines non-similar text data of a different type from the text classification data from the preset text classification dataset, and constructs dissimilar text data based on the non-similar text data. For example, the computing device determines non-similar text data of a different type from the text classification data "How to apply for a cash loan" from the preset text classification dataset, namely, text data not of the type "how to apply." The computing device randomly selects a certain number of non-similar text data from the non-similar text data to construct dissimilar text data. The computing device then constructs a text similarity dataset based on the similar text data and the dissimilar text data, wherein the ratio of the similar text data to the dissimilar text data in the text similarity dataset is 1:5.

[0051] Therefore, this technical solution generates a text similarity dataset based on a certain proportion of similar text data and dissimilar text data, and then trains the semantic vector extraction model through the text similarity dataset, so that the semantic vector extraction model not only learns whether the input text is similar, but also learns the true intention category of the input text, thereby enhancing the semantic extraction ability of the semantic vector extraction model.

[0052] Optionally, the dissimilar text data includes regular samples and difficult samples, and the operations of determining non-similar text data that is different from the text classification data from a preset text classification data set, and constructing dissimilar text data based on the non-similar text data, include: determining non-similar text data that is different from the text classification data from a preset text classification data set, and randomly constructing regular samples based on the non-similar text data; determining non-similar text data that is the nearest neighbor of the preset text classification data set and the text classification data through text clustering, and constructing difficult samples based on the non-similar text data; and constructing dissimilar text data based on regular samples and difficult samples, wherein the ratio of regular samples to difficult samples included in the dissimilar text data is 5:1.

[0053] Specifically, the dissimilar text data includes regular samples and difficult samples. The computing device determines non-similar text data that is different from the text classification data in a preset text classification data set, and randomly selects a certain number of non-similar text data from the non-similar text data to construct regular samples. The computing device then determines the non-similar text data that is the nearest neighbor of the preset text classification data set and the text classification data through text clustering, and constructs difficult samples based on the non-similar text data. Specifically, the computing device determines the non-similar text data that is the nearest neighbor of the text classification data with high similarity in the preset text classification data set through text clustering, and randomly selects a certain number of non-similar text data from the non-similar text data to construct difficult samples. The computing device then constructs dissimilar text data based on the regular samples and the difficult samples, wherein the ratio of regular samples to difficult samples included in the dissimilar text data is 5:1.

[0054] Therefore, this technical solution constructs a certain proportion of regular samples and difficult samples based on non-similar text data, thereby obtaining multiple samples for training the semantic vector extraction model. By using multiple samples, the semantic vector extraction model can learn whether the input text is similar and the true intention category of the input text, thereby enhancing the semantic extraction ability of the semantic vector extraction model.

[0055] Optionally, the operation of fine-tuning the multi-task classification network based on the text similarity dataset includes: inputting the text classification data and other text data in the text similarity dataset into the multi-task classification network respectively; the text classification data and other text data pass through the corresponding BERT language network in the multi-task classification network to output a first word vector and a second word vector; the first word vector and the second word vector pass through the corresponding attention mechanism network in the multi-task classification network to output a first text feature vector and a second text feature vector; the first text feature vector and the second text feature vector are respectively sent to the corresponding multi-classification fully connected layer to output a first classification result and a second classification result; the first text feature vector and the second text feature vector are sent together to the semantic matching layer to output a third classification result; and the multi-task classification network is iteratively trained according to the first classification result, the second classification result and the third classification result until the preset training conditions are met, then the training is stopped to obtain the multi-task classification network.

[0056] Specifically, refer to Figure 4As shown, the computing device inputs the text classification data and other text data in the text similarity dataset into the multi-task classification network respectively. The text classification data and other text data pass through the corresponding BERT language network in the multi-task classification network, and outputs a first word vector and a second word vector. For example, the computing device inputs the text classification data "Text1: How to apply for a cash loan" and "Text2: How to apply for a loan" in the text similarity dataset into the multi-task classification network respectively. The text lengths of "Text1: How to apply for a cash loan" and "Text2: How to apply for a loan" are L1 and L2 respectively. After passing through the corresponding BERT language network (12-layer encoder network) in the multi-task classification network, the output first word vector and second word vector are Vec1 and Vec2 respectively, and their dimensions are L1*768 and L2*768 respectively. After that, the first word vector and the second word vector pass through the corresponding attention mechanism network (Attention Layer) in the multi-task classification network, and output the first text feature vector and the second text feature vector. For example, the computing device applies weighted calculations to the first word vector Vec1 of dimension L1*768 and the second word vector Vec2 of dimension L2*768 through the corresponding attention layers in the multi-task classification network. The resulting weighted vectors are A1 and A2, with dimensions L1*1 and L2*1, respectively. At this point, the first and second text feature vectors output by the two attention layers are U and V, respectively, where U = Vev1*A1 and V = Vev2*A2. The computing device then feeds the first and second text feature vectors U and V into the corresponding multi-classification fully connected layers (Dense Layer), outputting the first classification result (i.e., the classification category of Text1) and the second classification result (i.e., the classification category of Text2). Simultaneously, the computing device feeds both the first and second text feature vectors U and V into the semantic matching layer (SimLayer), which performs semantic matching on the first and second text feature vectors U and V, and outputs a third classification result (i.e., whether Text1 and Text2 are similar). The computing device then iteratively trains the multi-task classification network according to the first classification result, the second classification result, and the third classification result until a preset training condition is met, and then stops training to obtain a multi-task classification network.

[0057] This technical solution, based on the conventional Siamese structural model, introduces a multi-task classification network. This allows the semantic vector extraction model to learn both the similarity of the input text and the true intent of the input text, thereby enhancing the semantic extraction capabilities of the multi-task classification network. Furthermore, an attention mechanism network (Attention Layer) is introduced to learn the importance of words and phrases, addressing the problem of semantic drift caused by redundant text.

[0058] Optionally, the semantic matching layer includes a semantic calculation layer and a binary classification layer, and the operation of sending the first text feature vector and the second text feature vector together into the semantic matching layer and outputting the third classification result includes: sending the first text feature vector and the second text feature vector together into the semantic calculation layer and outputting the third text feature vector with interactive information; and sending the third text feature vector into the binary classification layer and outputting the third classification result.

[0059] Specifically, the semantic matching layer (SimLayer) includes a semantic calculation layer and a binary classification layer. The computing device sends the first text feature vector and the second text feature vector into the semantic calculation layer together, and outputs a third text feature vector with interactive information. The computing device then sends the third text feature vector into the binary classification layer and outputs a third classification result. For example, the computing device sends the first text feature vector U and the second text feature vector V into the semantic calculation layer together, concatenates the first text feature vector U and the second text feature vector V into (U, V, |UV|, U*V), and outputs a third text feature vector S with interactive information, whose dimension is 1*3072. The computing device then sends the third text feature vector S into the binary classification layer and outputs a third classification result, that is, whether Text1 is similar to Text2. Therefore, the present technical solution utilizes the semantic calculation layer and the binary classification layer of the semantic matching layer (Sim Layer) to train the semantic matching ability of the multi-task classification network, thereby improving the semantic matching ability of the multi-task classification network.

[0060] Thus, according to the first aspect of this embodiment, the computing device uses the dialogue corpus of the same scene as the downstream task corresponding to the semantic vector as the first pre-training text, and performs secondary training on the trained BERT language network based on the first pre-training text to generate a BERT language network that focuses on nouns and verbs in the text, so that the BERT language network pays more attention to nouns and verbs with higher weights in text matching. Afterwards, the computing device obtains text classification data, generates a text similarity dataset based on the text classification data, and fine-tunes the multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model. Thus, this technical solution utilizes a multi-task classification network to fuse text classification and text similarity models. Feature learning is performed on the global representation and local comparison of the text, thereby improving the semantic understanding ability of the multi-task classification network.

[0061] Therefore, this technical solution uses dialogue materials to further pre-train the BERT language network, and the training task is changed to predicting words of specific parts of speech, so that the BERT language network has a better understanding of nouns and verbs with higher weights in similarity calculations. Compared with existing learning algorithms, this technical solution is based on the BERT language network and introduces multi-task training that integrates model similarity and text classification. By learning the connections and differences between different tasks, the quality of semantic representation is improved, and problems such as missing semantics and word order information will not occur. The internal interaction of the text makes the semantic expression unlimited, and also reduces the risk of model overfitting. The attention mechanism is introduced as a weak interaction in the representational semantic matching model to reduce the interference of redundant text, thereby improving the accuracy of the model. The technical effect of high quality of semantic representation and low limitation of semantic expression is achieved, thereby solving the technical problem of low quality of semantic representation and low limitation of semantic expression in the existing technology for semantic representation of text data.

[0062] In addition, according to the second aspect of this embodiment, a semantic vector representation method is provided, which is applicable to the semantic vector extraction model trained according to the first aspect of this embodiment. Figure 5 A schematic diagram of the process is shown in FIG. Figure 5 As shown, the method includes:

[0063] S502: Obtaining the text to be recognized; and

[0064] S504: Input the text to be recognized into a semantic vector extraction model to obtain a semantic vector of the text to be recognized.

[0065] Specifically, the computing device obtains the text to be recognized and inputs the text to be recognized into a semantic vector extraction model to obtain a semantic vector for the text to be recognized. For example, the text to be recognized is a customer query, "Can you please stop making random calls? Can you please reply to me quickly? I got my salary today and I want to pay back all the money I owed." Obviously, only the last short sentence in the customer query is important, and the previous sentences are meaningless. Since the semantic vector extraction model trained in the first aspect of this embodiment not only has a better understanding of nouns and verbs with higher weights in the text, but also has the semantic understanding ability of a multi-task classification network, after the text to be recognized is input into the semantic vector extraction model, the semantic vector extraction model can pay more attention to the key content in long and difficult sentences, thereby reducing the semantic deviation caused by redundant long sentences. That is, after the text to be recognized is input into the semantic vector extraction model, the model can accurately determine that "I want to pay back all the money" is a key phrase in the content of the text to be recognized, and further calculate the semantic vector of "I want to pay back all the money". Therefore, the semantic vector extraction model trained using the method of the first aspect of this embodiment in the second aspect of this embodiment can accurately extract semantic vectors from the text to be recognized, and the extracted semantic vectors do not suffer from the problem of low semantic representation quality or limited semantic expression. This solves the technical problems of low semantic representation quality and limited semantic expression in existing methods of semantically representing text data.

[0066] Optionally, the method also includes: matching the semantic vector of the text to be recognized with the semantic vector in a preset FAQ library, wherein the semantic vectors of each text data in the FAQ library are extracted in advance using a semantic vector extraction model and stored in the FAQ library; and based on the result of the matching calculation, determining the target text that matches the text to be recognized from the FAQ library.

[0067] Specifically, for example, the computing device calculates the semantic vectors in the FAQ library offline in advance. When used online, it only needs to infer the semantic vectors of the text to be recognized in real time, and then by calculating the matching degree between the semantic vectors, the topn closest to the text to be recognized can be determined from the FAQ library. This makes the semantic recall speed very fast. The semantic vectors of each text data in the FAQ library are extracted in advance using the semantic vector extraction model and stored in the FAQ library. Therefore, after inferring the semantic vector of the text to be recognized, the vector matching tool can be used to directly calculate the topn. There is no need to use real-time BERT language network inference calculation, which reduces the inference steps and greatly improves the inference speed.

[0068] In addition, reference Figure 1As shown, according to a third aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0069] Therefore, according to this embodiment, this technical solution uses dialogue materials to further pre-train the BERT language network, and the training task is changed to predicting words of a specific part of speech, so that the BERT language network has a better understanding of nouns and verbs with higher weights in similarity calculations. Compared with existing learning algorithms, this technical solution is based on the BERT language network and introduces multi-task training that integrates model similarity and text classification. By learning the connections and differences between different tasks, the quality of semantic representation is improved, and problems such as missing semantics and word order information will not occur. The internal interaction of the text makes the semantic expression unlimited, and also reduces the risk of model overfitting. The attention mechanism is introduced as a weak interaction in the representational semantic matching model to reduce the interference of redundant text, thereby improving the accuracy of the model. The technical effect of high quality of semantic representation and low limitation of semantic expression is achieved, thereby solving the technical problem of low quality of semantic representation and low limitation of semantic expression in the existing technology for semantic representation of text data.

[0070] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0071] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0072] Example 2

[0073] Figure 7 The apparatus 700 for training the semantic vector extraction model according to the first aspect of this embodiment is shown. The apparatus 700 corresponds to the method according to the first aspect of embodiment 1. Figure 7 As shown, the device 700 includes: a first generation module 710, which is used to obtain a dialogue corpus of the same scene as the downstream task corresponding to the semantic vector as a first pre-training text, and based on the first pre-training text, perform secondary training on the trained BERT language network to generate a BERT language network that focuses on nouns and verbs in the text; a second generation module 720, which is used to obtain text classification data and generate a text similarity dataset based on the text classification data, wherein the text classification data is classified text data, and the text similarity dataset consists of the text classification data and other text data with different degrees of similarity to the text classification data; and a third generation module 730, which is used to fine-tune the multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model, wherein the multi-task classification network includes two BERT language networks after secondary training, two attention mechanism networks, two multi-classification fully connected layers and one semantic matching layer, and the semantic vector extraction model includes the BERT language network and the attention mechanism network in the multi-task classification network generated at the end of fine-tuning training.

[0074] Optionally, the first generation module 710 includes: a first generation submodule for processing the first pre-trained text to generate an input text in a single-round dialogue format; a marking submodule for performing word segmentation marking and part-of-speech marking on the input text, wherein part-of-speech marking on the input text includes noun marking and verb marking on the input text; a mask replacement submodule for replacing a predetermined number of nouns and verbs in the input text with masks based on the results of word segmentation marking and part-of-speech marking; and a second generation submodule for performing secondary training on the trained BERT language network based on the masked input text, so that the BERT language network predicts the masked words, thereby generating a BERT language network that focuses on the nouns and verbs in the text.

[0075] Optionally, other text data having different degrees of similarity with the text classification data include similar text data and dissimilar text data, and the second generation module 720 includes: a first construction submodule, used to determine similar text data of the same type as the text classification data from a preset text classification data set, and randomly construct similar text data based on the similar text data, wherein the preset text classification data set is composed of text classification data of different categories; a second construction submodule, used to determine non-similar text data of a different type from the text classification data from the preset text classification data set, and construct dissimilar text data based on the non-similar text data; and a third construction submodule, used to construct a text similarity data set based on the similar text data and the dissimilar text data, wherein the ratio of similar text data to dissimilar text data included in the text similarity data set is 1:5.

[0076] Optionally, the dissimilar text data includes regular samples and difficult samples, and the second construction submodule includes: a first construction unit, used to determine non-similar text data that is different from the text classification data from a preset text classification data set, and randomly construct regular samples based on the non-similar text data; a second construction unit, used to determine the non-similar text data that is the nearest neighbor of the preset text classification data set and the text classification data through text clustering, and construct difficult samples based on the non-similar text data; and a third construction unit, used to construct dissimilar text data based on regular samples and difficult samples, wherein the ratio of regular samples to difficult samples included in the dissimilar text data is 5:1.

[0077] Optionally, the third generation module 730 includes: an input submodule, which is used to input the text classification data and other text data in the text similarity dataset into the multi-task classification network respectively; a first output submodule, which is used to pass the text classification data and other text data through the corresponding BERT language network in the multi-task classification network to output the first word vector and the second word vector; a second output submodule, which is used to pass the first word vector and the second word vector through the corresponding attention mechanism network in the multi-task classification network to output the first text feature vector and the second text feature vector; a third output submodule, which is used to send the first text feature vector and the second text feature vector into the corresponding multi-classification fully connected layer respectively, and output the first classification result and the second classification result; a fourth output submodule, which is used to send the first text feature vector and the second text feature vector together into the semantic matching layer, and output the third classification result; and an iterative training submodule, which is used to iteratively train the multi-task classification network according to the first classification result, the second classification result and the third classification result until the preset training conditions are met, then stop training to obtain the multi-task classification network.

[0078] Optionally, the semantic matching layer includes a semantic calculation layer and a binary classification layer, and the third output submodule includes: a first output unit, used to send the first text feature vector and the second text feature vector into the semantic calculation layer together, and output a third text feature vector with interactive information; and a second output unit, used to send the third text feature vector into the binary classification layer, and output a third classification result.

[0079] also, Figure 8 The semantic vector representation device 800 according to the second aspect of this embodiment is shown, which is applicable to the semantic vector extraction model trained in the first aspect of this embodiment. The device 800 corresponds to the method described in the second aspect of embodiment 1. Figure 8 As shown, the apparatus 800 includes: a first acquisition module 810 for acquiring a text to be recognized; and a second acquisition module 820 for inputting the text to be recognized into a semantic vector extraction model to acquire a semantic vector of the text to be recognized.

[0080] Optionally, the device 800 also includes a matching calculation module for matching the semantic vector of the text to be recognized with the semantic vector in a preset FAQ library, wherein the semantic vectors of each text data in the FAQ library are extracted in advance using a semantic vector extraction model and stored in the FAQ library; and a determination module for determining the target text that matches the text to be recognized from the FAQ library based on the result of the matching calculation.

[0081] Therefore, according to this embodiment, this technical solution uses dialogue materials to further pre-train the BERT language network, and the training task is changed to predicting words of a specific part of speech, so that the BERT language network has a better understanding of nouns and verbs with higher weights in similarity calculations. Compared with existing learning algorithms, this technical solution is based on the BERT language network and introduces multi-task training that integrates model similarity and text classification. By learning the connections and differences between different tasks, the quality of semantic representation is improved, and problems such as missing semantics and word order information will not occur. The internal interaction of the text makes the semantic expression unlimited, and also reduces the risk of model overfitting. The attention mechanism is introduced as a weak interaction in the representational semantic matching model to reduce the interference of redundant text, thereby improving the accuracy of the model. The technical effect of high quality of semantic representation and low limitation of semantic expression is achieved, thereby solving the technical problem of low quality of semantic representation and low limitation of semantic expression in the existing technology for semantic representation of text data.

[0082] Example 3

[0083] Figure 9 The apparatus 900 for training the semantic vector extraction model according to the first aspect of this embodiment is shown. The apparatus 900 corresponds to the method according to the first aspect of embodiment 1. Figure 9As shown, the device 900 includes: a first processor 910; and a first memory 920, connected to the first processor 910, for providing the first processor 910 with instructions for processing the following processing steps: obtaining a dialogue corpus of the same scene as the downstream task corresponding to the semantic vector as a first pre-training text, and based on the first pre-training text, performing secondary training on the trained BERT language network to generate a BERT language network that focuses on nouns and verbs in the text; obtaining text classification data, and generating a text similarity dataset based on the text classification data, wherein the text classification data is classified text data, and the text similarity dataset consists of the text classification data and other text data with different degrees of similarity to the text classification data; and fine-tuning the multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model, wherein the multi-task classification network includes two BERT language networks after secondary training, two attention mechanism networks, two multi-classification fully connected layers and one semantic matching layer, and the semantic vector extraction model includes the BERT language network and the attention mechanism network in the multi-task classification network generated at the end of fine-tuning training.

[0084] Optionally, based on the first pre-trained text, the trained BERT language network is retrained to generate a BERT language network that focuses on nouns and verbs in the text, including: processing the first pre-trained text to generate an input text in a single-round dialogue format; performing word segmentation tagging and part-of-speech tagging on the input text, wherein the part-of-speech tagging of the input text includes noun tagging and verb tagging on the input text; based on the results of the word segmentation tagging and part-of-speech tagging, using a mask to replace a predetermined number of nouns and verbs in the input text; and based on the masked input text, the trained BERT language network is retrained to enable the BERT language network to predict the masked words, thereby generating a BERT language network that focuses on the nouns and verbs in the text.

[0085] Optionally, the other text data having different degrees of similarity with the text classification data include similar text data and dissimilar text data, and the operation of generating a text similarity dataset based on the text classification data includes: determining similar text data of the same type as the text classification data from a preset text classification dataset, and randomly constructing similar text data based on the similar text data, wherein the preset text classification dataset is composed of text classification data of different categories; determining non-similar text data of a different type from the text classification data from the preset text classification dataset, and constructing dissimilar text data based on the non-similar text data; and constructing a text similarity dataset based on the similar text data and the dissimilar text data, wherein the ratio of similar text data to dissimilar text data included in the text similarity dataset is 1:5.

[0086] Optionally, the dissimilar text data includes regular samples and difficult samples, and the operations of determining non-similar text data that is different from the text classification data from a preset text classification data set, and constructing dissimilar text data based on the non-similar text data, include: determining non-similar text data that is different from the text classification data from a preset text classification data set, and randomly constructing regular samples based on the non-similar text data; determining non-similar text data that is the nearest neighbor of the preset text classification data set and the text classification data through text clustering, and constructing difficult samples based on the non-similar text data; and constructing dissimilar text data based on regular samples and difficult samples, wherein the ratio of regular samples to difficult samples included in the dissimilar text data is 5:1.

[0087] Optionally, the operation of fine-tuning the multi-task classification network based on the text similarity dataset includes: inputting the text classification data and other text data in the text similarity dataset into the multi-task classification network respectively; the text classification data and other text data pass through the corresponding BERT language network in the multi-task classification network to output a first word vector and a second word vector; the first word vector and the second word vector pass through the corresponding attention mechanism network in the multi-task classification network to output a first text feature vector and a second text feature vector; the first text feature vector and the second text feature vector are respectively sent to the corresponding multi-classification fully connected layer to output a first classification result and a second classification result; the first text feature vector and the second text feature vector are sent together to the semantic matching layer to output a third classification result; and the multi-task classification network is iteratively trained according to the first classification result, the second classification result and the third classification result until the preset training conditions are met, then the training is stopped to obtain the multi-task classification network.

[0088] Optionally, the semantic matching layer includes a semantic calculation layer and a binary classification layer, and the operation of sending the first text feature vector and the second text feature vector together into the semantic matching layer and outputting the third classification result includes: sending the first text feature vector and the second text feature vector together into the semantic calculation layer and outputting the third text feature vector with interactive information; and sending the third text feature vector into the binary classification layer and outputting the third classification result.

[0089] also, Figure 10 The semantic vector representation device 1000 according to the second aspect of this embodiment is shown, which is applicable to the semantic vector extraction model trained in the first aspect of this embodiment. The device 1000 corresponds to the method described in the second aspect of embodiment 1. Figure 10As shown, the device 1000 includes: a second processor 1010; and a second memory 1020, which is connected to the second processor 1010 and is used to provide the second processor 1010 with instructions for processing the following processing steps: obtaining the text to be recognized; and inputting the text to be recognized into a semantic vector extraction model to obtain the semantic vector of the text to be recognized.

[0090] Optionally, the second memory 1020 is also used to provide the second processor 1010 with instructions for processing the following processing steps: matching the semantic vector of the text to be recognized with the semantic vector in a preset FAQ library, wherein the semantic vectors of each text data in the FAQ library are extracted in advance using a semantic vector extraction model and stored in the FAQ library; and determining the target text that matches the text to be recognized from the FAQ library based on the result of the matching calculation.

[0091] Therefore, according to this embodiment, this technical solution uses dialogue materials to further pre-train the BERT language network, and the training task is changed to predicting words of a specific part of speech, so that the BERT language network has a better understanding of nouns and verbs with higher weights in similarity calculations. Compared with existing learning algorithms, this technical solution is based on the BERT language network and introduces multi-task training that integrates model similarity and text classification. By learning the connections and differences between different tasks, the quality of semantic representation is improved, and problems such as missing semantics and word order information will not occur. The internal interaction of the text makes the semantic expression unlimited, and also reduces the risk of model overfitting. The attention mechanism is introduced as a weak interaction in the representational semantic matching model to reduce the interference of redundant text, thereby improving the accuracy of the model. The technical effect of high quality of semantic representation and low limitation of semantic expression is achieved, thereby solving the technical problem of low quality of semantic representation and low limitation of semantic expression in the existing technology for semantic representation of text data.

[0092] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0093] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0095] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0096] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0098] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A training method for a semantic vector extraction model, characterized in that: include: Obtaining a dialogue corpus in the same scenario as the downstream task corresponding to the semantic vector as a first pre-trained text, and performing a secondary training on the trained BERT language network based on the first pre-trained text to generate a BERT language network that focuses on nouns and verbs in the text; Acquire text classification data, and generate a text similarity dataset based on the text classification data, wherein the text classification data is classified text data, and the text similarity dataset consists of the text classification data and other text data having different degrees of similarity with the text classification data; as well as Based on the text similarity dataset, the multi-task classification network is fine-tuned to generate a semantic vector extraction model, wherein the multi-task classification network includes two BERT language networks after secondary training, two attention mechanism networks, two multi-classification fully connected layers and a semantic matching layer, and the semantic vector extraction model includes the BERT language network and attention mechanism network in the multi-task classification network generated at the end of fine-tuning training.

2. The method according to claim 1, characterized in that The operation of performing secondary training on the trained BERT language network based on the first pre-trained text to generate a BERT language network that focuses on nouns and verbs in the text includes: Processing the first pre-trained text to generate input text in a single-turn dialogue format; Performing word segmentation marking and part-of-speech marking on the input text, wherein performing part-of-speech marking on the input text includes performing noun marking and verb marking on the input text; Using masks to replace a predetermined number of nouns and verbs in the input text according to the results of the word segmentation tags and the part-of-speech tags; and Based on the masked input text, the trained BERT language network is retrained so that the BERT language network predicts the masked words, thereby generating a BERT language network that focuses on nouns and verbs in the text.

3. The method according to claim 1, characterized in that The other text data having different degrees of similarity with the text classification data include similar text data and dissimilar text data, and the operation of generating a text similarity dataset based on the text classification data includes: Determining similar text data of the same type as the text classification data from a preset text classification data set, and randomly constructing the similar text data based on the similar text data, wherein the preset text classification data set is composed of text classification data of different categories; Determining non-similar text data that is different from the text classification data from the preset text classification data set, and constructing the dissimilar text data based on the non-similar text data; and The text similarity dataset is constructed based on the similar text data and the dissimilar text data, wherein the ratio of the similar text data to the dissimilar text data included in the text similarity dataset is 1:

5.

4. The method according to claim 3, characterized in that The dissimilar text data includes regular samples and difficult samples, and the operations of determining non-similar text data that is different from the text classification data from the preset text classification data set, and constructing the dissimilar text data based on the non-similar text data include: Determining non-similar text data that is different from the text classification data from the preset text classification data set, and randomly constructing the regular sample based on the non-similar text data; Determine the preset text classification data set and the non-similar text data that are nearest neighbors of the text classification data by text clustering, and construct the difficult sample based on the non-similar text data; and The dissimilar text data is constructed based on the regular samples and the difficult samples, wherein the ratio of the regular samples to the difficult samples included in the dissimilar text data is 5:

1.

5. The method according to claim 1, wherein The operation of fine-tuning the multi-task classification network based on the text similarity dataset includes: Inputting the text classification data and the other text data in the text similarity dataset into the multi-task classification network respectively; The text classification data and the other text data are passed through the corresponding BERT language network in the multi-task classification network to output a first word vector and a second word vector; The first word vector and the second word vector are passed through the corresponding attention mechanism network in the multi-task classification network to output a first text feature vector and a second text feature vector; Sending the first text feature vector and the second text feature vector to the corresponding multi-classification fully connected layer respectively, and outputting a first classification result and a second classification result; Sending the first text feature vector and the second text feature vector together to the semantic matching layer, and outputting a third classification result; and According to the first classification result, the second classification result and the third classification result, the multi-task classification network is iteratively trained until a preset training condition is met, and then the training is stopped to obtain the multi-task classification network.

6. The method according to claim 5, characterized in that The semantic matching layer includes a semantic calculation layer and a binary classification layer, and the operation of sending the first text feature vector and the second text feature vector together to the semantic matching layer and outputting a third classification result includes: Sending the first text feature vector and the second text feature vector together to the semantic calculation layer, and outputting a third text feature vector with interaction information; and The third text feature vector is sent to the binary classification layer, and the third classification result is output.

7. A semantic vector representation method, applicable to the semantic vector extraction model trained according to claims 1-6, the method comprising: Get the text to be recognized; as well as The text to be recognized is input into the semantic vector extraction model to obtain the semantic vector of the text to be recognized.

8. The method according to claim 7, characterized in that Also includes: Matching and calculating the semantic vector of the text to be recognized with the semantic vectors in a preset FAQ library, wherein the semantic vectors of each text data in the FAQ library are extracted in advance using the semantic vector extraction model and stored in the FAQ library; and According to the result of the matching calculation, a target text that matches the text to be recognized is determined from the FAQ library.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 8.

10. A training device for a semantic vector extraction model, characterized in that: include: A first generation module is configured to obtain a dialogue corpus corresponding to the semantic vector and belonging to the same scenario as the downstream task as a first pre-training text, and to perform secondary training on the trained BERT language network based on the first pre-training text to generate a BERT language network that focuses on nouns and verbs in the text; a second generating module, configured to obtain text classification data and generate a text similarity dataset based on the text classification data, wherein the text classification data is classified text data, and the text similarity dataset consists of the text classification data and other text data having different degrees of similarity with the text classification data; as well as The third generation module is used to fine-tune the multi-task classification network based on the text similarity dataset to generate a semantic vector extraction model, wherein the multi-task classification network includes two BERT language networks after secondary training, two attention mechanism networks, two multi-classification fully connected layers and one semantic matching layer, and the semantic vector extraction model includes the BERT language network and attention mechanism network in the multi-task classification network generated at the end of fine-tuning training.

Images