Text vectorization model training method, text processing method, device and equipment

Abstract

The present application provides a training method, a text processing method, an apparatus and a device for a text vectorization model; the method includes: clustering the answer texts in multiple groups of question and answer texts to obtain multiple answer text clusters, and constructing a question text cluster corresponding to each answer text cluster; obtaining at least one positive question sample different from the first question text from the first question text cluster where the first question text is located, wherein the first question text is any question text; obtaining at least one negative question sample from the second question text cluster; training the first text vectorization model based on multiple first question texts and the positive question samples and negative question samples corresponding to the multiple first question texts, and obtaining the trained first text vectorization model. Through the present application, the text representation ability of the first text vectorization model can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, a text processing method, a device and equipment for a text vectorization model. Background Art

[0002] In related technologies, open source models such as the BAAI General Embedding (BGE) are used to obtain vectorized representations of texts. In order to further improve the performance of the model for vectorization, a batch of texts of target scenes and training data of positive and negative examples corresponding to the texts are annotated using a manual annotation scheme, and further fine-tuning is performed based on the existing open source model. However, manual annotation requires a lot of manpower and time.

[0003] Related technologies lack solutions to effectively improve the model's text representation capabilities. Summary of the invention

[0004] The embodiments of the present application provide a training method, a text processing method, an apparatus and a device for a text vectorization model, which can improve the text representation capability of a first text vectorization model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application embodiment provides a method for training a text vectorization model, the method comprising:

[0007] Clustering the answer texts in the multiple groups of question and answer texts to obtain multiple answer text clusters, and constructing a question text cluster corresponding to each of the answer text clusters, wherein each group of the question and answer texts includes a question text and the answer text corresponding to the question text;

[0008] Acquire at least one question positive sample different from the first question text from the first question text cluster where the first question text is located, wherein the first question text is any one of the question texts;

[0009] Acquire at least one question negative sample from a second question text cluster, wherein the second question text cluster is any question text cluster different from the first question text cluster;

[0010] Based on the plurality of first question texts, the question positive samples and the question negative samples respectively corresponding to the plurality of first question texts, a first text vectorization model is trained to obtain a trained first text vectorization model.

[0011] The present application provides a text processing method, the method comprising:

[0012] Get multiple object question texts;

[0013] Vectorizing the multiple object question texts through a first text vectorization model to obtain multiple object question text vectors, wherein the first text vectorization model is trained by the text vectorization model training method provided in an embodiment of the present application;

[0014] Clustering the multiple object question text vectors to obtain at least one object question text cluster;

[0015] The answer pre-associated with each of the object question text clusters is obtained as an answer in response to the object question text in each of the object question text clusters.

[0016] The present application embodiment provides a training device for a text vectorization model, including:

[0017] A data processing module, used for clustering answer texts in multiple groups of question and answer texts to obtain multiple answer text clusters, and constructing a question text cluster corresponding to each answer text cluster, wherein each group of question and answer texts includes a question text and the answer text corresponding to the question text;

[0018] The data processing module is further used to obtain at least one question positive sample different from the first question text from the first question text cluster where the first question text is located, wherein the first question text is any one of the question texts;

[0019] The data processing module is further used to obtain at least one negative question sample from a second question text cluster, wherein the second question text cluster is any question text cluster different from the first question text cluster;

[0020] A training module is used to train a first text vectorization model based on a plurality of the first question texts, the question positive samples and the question negative samples respectively corresponding to the plurality of the first question texts, so as to obtain a trained first text vectorization model.

[0021] The present application provides a text processing device, including:

[0022] A data acquisition module is used to acquire multiple object question texts;

[0023] A data processing module, used to perform vectorization processing on the multiple object question texts through a first text vectorization model to obtain multiple object question text vectors, wherein the first text vectorization model is trained by the text vectorization model training method provided in an embodiment of the present application;

[0024] The data processing module is further used to perform clustering processing on the multiple object question text vectors to obtain at least one object question text cluster;

[0025] The data processing module is further used to obtain an answer pre-associated with each of the object question text clusters as an answer in response to the object question text in each of the object question text clusters.

[0026] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0027] A memory for storing computer executable instructions or computer programs;

[0028] The processor is used to implement the training method of the text vectorization model provided in the embodiment of the present application, or implement the text processing method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0029] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which are used to implement the training method of the text vectorization model provided in the embodiment of the present application, or implement the text processing method provided in the embodiment of the present application when executed by a processor.

[0030] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the training method of the text vectorization model provided in the embodiment of the present application is implemented, or the text processing method provided in the embodiment of the present application is implemented.

[0031] The embodiments of the present application have the following beneficial effects:

[0032] The answer texts are clustered by taking the answer text corresponding to each question text as an anchor point. After obtaining the answer text cluster, the question text cluster corresponding to each answer text cluster is constructed according to the clustering result of the answer text. This makes full use of the correlation between the question text and the answer text, avoids the problem of poor effect of directly clustering the question text due to the diversity and dispersion of the question text expression, improves the clustering effect of the question text, and thus improves the quality of the question positive samples and the question negative samples obtained through the intra-cluster and inter-cluster relationships of the question text, and finally improves the text representation ability of the first text vectorization model trained based on the first question text and the question positive samples and the question negative samples corresponding to the first question text. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural diagram of the text processing system architecture provided by an embodiment of the present application;

[0034] Figure 2A is a first structural diagram of a server provided in an embodiment of the present application;

[0035] Figure 2B is a second structural diagram of a server provided in an embodiment of the present application;

[0036] Figure 3A It is a first flow chart of the training method of the text vectorization model provided in the embodiment of the present application;

[0037] Figure 3B It is a second flow chart of the training method of the text vectorization model provided in the embodiment of the present application;

[0038] Figure 3C 3 is a schematic diagram of a third process of the training method of the text vectorization model provided in the embodiment of the present application;

[0039] Figure 3D It is a fourth flow chart of the training method of the text vectorization model provided in the embodiment of the present application;

[0040] Figure 3E It is a fifth flow chart of the training method of the text vectorization model provided in the embodiment of the present application;

[0041] Figure 3F It is a sixth flow chart of the training method of the text vectorization model provided in the embodiment of the present application;

[0042] Figure 3G 7 is a schematic diagram of a seventh flow chart of a method for training a text vectorization model provided in an embodiment of the present application;

[0043] Figure 3H It is an eighth flow chart of the training method of the text vectorization model provided in the embodiment of the present application;

[0044] Figure 4 It is a flowchart of a text processing method provided in an embodiment of the present application;

[0045] Figure 5 It is a schematic diagram of the application process of the text processing method provided in the embodiment of the present application;

[0046] Figure 6 It is a first principle diagram of the training method of the text vectorization model provided in the embodiment of the present application;

[0047] Figure 7 It is a second principle diagram of the training method of the text vectorization model provided in the embodiment of the present application;

[0048] Figure 8 It is a schematic diagram of the interface of the text processing method provided in the embodiment of the present application.

[0049] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0051] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0053] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0054] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0055] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0058] 2) Text processing interface, which is used to provide human-computer interaction interface / interface for displaying text processing information.

[0059] For example, graphical user interface (GUI) displays, such as augmented reality (AR) interface, virtual reality (VR) interface, voice user interface (VUI), interactive projection interface (using projection technology to display information on a plane), eye movement detection interface (interface controlled by detecting the user's line of sight), holographic interface (three-dimensional hologram formed by projecting images through holographic projection technology, and stereoscopic images can be seen without wearing special glasses), multimodal interface (interface that combines multiple interaction methods, such as touch, vision, hearing, etc.), brain-machine interface (Brain-Machine Interface, BMI) interface, etc.

[0060] 3) Answer text cluster refers to a set of answer texts that are similar or related in content, semantics or style. The clustering algorithm groups the answer texts in multiple sets of question and answer texts according to their similarity, and each group forms a cluster.

[0061] 4) Question text cluster refers to a question text set consisting of the question text corresponding to each answer text in the answer text cluster.

[0062] 5) Text vectorization model, which is used to convert text into vector representations in a high-dimensional space. These vectors can capture the deep semantic information of the text.

[0063] 6) Contrastive Learning is an unsupervised learning method that aims to learn data representation by comparing the similarities between different samples. Contrastive learning can be used for tasks such as image retrieval and text matching in the field of natural language processing.

[0064] 7) Domain refers to a branch of specific knowledge or activities. In the context of data science and big data, "domain" can include industries or professional fields (such as medical, gaming, education, law, etc.), data types or formats (such as text, images, audio, etc.), application fields (such as data analysis, machine learning model training, etc.) and knowledge fields (such as physics, biology, etc.).

[0065] 8) A token is the smallest processing unit into which text data is segmented. It can be a word, phrase, punctuation mark, character sequence, or any other text unit that has meaning in a specific context.

[0066] 9) Transformer refers to a time series model based on the self-attention mechanism. The encoder part can effectively encode time series information. Its processing ability of time series information is much better than that of the long short-term memory network (LSTM), and its speed is fast. It is widely used in natural language processing, computer vision, machine translation, speech recognition and other fields.

[0067] The related technologies have the following limitations:

[0068] 1) By using open source models, such as the general semantic vector model (BGE), to obtain the vectorized representation of text, although it can be quickly launched at low cost, the vectorization effect is not ideal due to the certain gap between the training text used by the general model and the text distribution of the target business scenario. For example, the training data of the general model is often constructed by the search text searched by Internet users and its corresponding web documents. The user's search text is very scattered and involves various fields; while for the target business scenario (such as human-machine collaborative customer service scenario), user question texts are often concentrated in certain small areas. Therefore, the solution of using the general semantic vector model to directly vectorize the text still has a lot of room for improvement.

[0069] 2) By introducing annotated data in the field of the target business scenario, further fine-tuning based on the existing open source model can improve model performance. However, to achieve a more ideal effect, about 5,000 to 10,000 training data are required in a single field, and manual labeling consumes a lot of manpower and time. If there are multiple different business scenarios that need to be fine-tuned at the same time, it will consume a lot of resources and it will be difficult to quickly apply to the target business scenario.

[0070] The embodiments of the present application provide a training method, a text processing method, an apparatus, a device, a computer-readable storage medium, and a computer program product for a text vectorization model, which can improve the text representation capability of a first text vectorization model. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The device provided by the embodiments of the present application can be implemented as various types of terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as a server.

[0071] See also Figure 1 , Figure 1 is a structural diagram of the text processing system architecture provided in an embodiment of the present application, Figure 1 The server 100, the terminal device 200 and the network 300 are involved. The terminal device 200 is connected to the server 100 via the network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0072] In some embodiments, the embodiments of the present application can be implemented collaboratively by a server and a terminal device. For example, the terminal device 200 sends multiple object question texts to the server 100, and the server 100 receives the multiple object question texts, performs text processing on the multiple object question texts through the text processing method provided in the embodiment of the present application, obtains at least one object question text cluster, obtains the answer pre-associated with each object question text cluster, and sends the answer to the object question text in each object question text cluster to the terminal device 200. Here, the multiple object question texts are processed to obtain at least one object question text cluster, which is implemented by the first text vectorization model trained by the training method of the text vectorization model provided in the embodiment of the present application.

[0073] The training method of the text vectorization model provided in the embodiment of the present application can be applied to various scenarios that require text processing, such as intelligent customer service scenarios, text retrieval scenarios, text recommendation scenarios, etc., which are illustrated by examples below.

[0074] 1) In question-answering systems (such as intelligent customer service, online assistants, etc.), text vectorization models can be used to understand users’ questions and retrieve the most relevant answers from the knowledge base or perform corresponding tasks.

[0075] 2) In recommendation systems, video platforms or music streaming services can use text vectorization models to analyze user comments and retrieve text and other data to recommend videos or music that users may be interested in.

[0076] 3) In search engine systems, when users enter queries into search engines, text vectorization models can be used to understand the query intent and retrieve the most relevant results from the indexed documents.

[0077] In other embodiments, the embodiments of the present application can be implemented by a terminal device alone. The terminal device 200 sends a model acquisition request to the server 100, and the server 100 receives the model acquisition request and sends a first text vectorization model used for performing the text processing method provided in the embodiment of the present application to the terminal device 200. The first text vectorization model is obtained by training the server 100 through the text vectorization model training method provided in the embodiment of the present application. The terminal device 200 receives the first text vectorization model sent by the server and downloads it locally, processes multiple object question texts through the text processing method provided in the embodiment of the present application, obtains at least one object question text cluster, and obtains the answer pre-associated with each object question text cluster as an answer to the object question text in each object question text cluster.

[0078] Here, the server 100 may be a single server. In this case, the training method and text processing method of the text vectorization model provided in the embodiment of the present application may be implemented by the same server. The server 100 may also be a cluster of servers. In the case where the server 100 is a server cluster, the training method and text processing method of the text vectorization model provided in the embodiment of the present application may be implemented by different servers, which is not limited in the embodiment of the present application.

[0079] Take the server for training text vectorization model as an example, see Figure 2A , Figure 2A is a first structural diagram of a server provided in an embodiment of the present application, Figure 2A The server 100-1 shown includes: at least one processor 110-1, a memory 130-1 and at least one network interface 120-1. The various components in the server 100-1 are coupled together via a bus system 140-1. It is understood that the bus system 140-1 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 140-1 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140-1 is not described in detail. Figure 2A Various buses are labeled as bus system 140 - 1 .

[0080] Processor 110 - 1 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0081] Memory 130-1 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 130-1 may optionally include one or more storage devices that are physically remote from processor 110-1.

[0082] The memory 130-1 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 130-1 described in the embodiment of the present application is intended to include any suitable type of memory.

[0083] In some embodiments, the memory 130 - 1 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0084] Operating system 131-1, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0085] A network communication module 132-1, used to reach other electronic devices via one or more (wired or wireless) network interfaces 120-1, exemplary network interfaces 120-1 include: Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB), etc.;

[0086] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2A The training device 133 of the text vectorization model stored in the memory 130-1 is shown, which can be software in the form of a program and a plug-in, including the following software modules: a data processing module 1331 and a training module 1332. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0087] For example, using the server for text processing, see Figure 2B , Figure 2B is a second structural diagram of a server provided in an embodiment of the present application, Figure 2BThe server 100-2 shown includes: at least one processor 110-2, a memory 130-2 and at least one network interface 120-2. The various components in the server 100-2 are coupled together via a bus system 140-2. It is understood that the bus system 140-2 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 140-2 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140-2 is not described in detail. Figure 2B Various buses are labeled as bus system 140-2. For the detailed description of processor 110-2 and memory 130-2, please refer to the above, which will not be repeated here.

[0088] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2B The text processing device 134 stored in the memory 130-2 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data acquisition module 1341 and a data processing module 1342. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0089] In some embodiments, the terminal device or server can implement the training method and text processing method of the text vectorization model provided in the embodiments of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native application (APPlication, APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to the browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0090] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method and text processing method of the text vectorization model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0091] The following will combine the exemplary application and implementation of the server provided in the embodiment of the present application, taking the server as the execution subject, to illustrate the training method of the text vectorization model provided in the embodiment of the present application. Figure 6 , by clustering the answer texts in multiple groups of question and answer texts, multiple answer text clusters are obtained, based on the one-to-one correspondence between the answer texts and the question texts, a question text cluster corresponding to each answer text cluster is constructed to obtain multiple question text clusters (corresponding to step 101 below), and a positive question sample of the first question text is obtained by random sampling within the cluster based on the multiple question text clusters (corresponding to step 102 below), and a negative question sample of the first question text is obtained by random sampling after cluster filtering based on the multiple question text clusters (corresponding to step 103 below), the first question text is any question text, and a training sample is constructed based on each first question text, each positive question sample corresponding to the first question text, and a negative question sample (that is, each training sample includes a first question text, a positive question sample corresponding to the first question text, and a negative question sample), and a first text vectorization model is trained based on the training samples (corresponding to step 104 below).

[0092] See also Figure 3A , Figure 3A This is a first flow chart of the training method of the text vectorization model provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0093] In step 101, the answer texts in multiple groups of question and answer texts are clustered to obtain multiple answer text clusters, and a question text cluster corresponding to each answer text cluster is constructed, wherein each group of question and answer text includes a question text and an answer text corresponding to the question text.

[0094] In some embodiments, multiple groups of question-answer texts are sets containing multiple question-answer pairs, each question-answer pair including a question text and an answer text corresponding to the question text. For example, a question-answer pair may include: question text: "What is the weather like today?" and answer text: "It is sunny today, and the temperature is about 25 degrees Celsius".

[0095] For example, see Figure 7 , multiple sets of question-answer texts include question texts (corresponding to Figure 7 Q 1 ,q 2 ,q 3 , …, q N ) and the answer text corresponding to each question text (corresponding to Figure 7 a 1 , a 2 , a 3 , …, a N ).

[0096] In some embodiments, clustering the answer texts in multiple groups of question and answer texts to obtain multiple answer text clusters is achieved through a second text vectorization model, the number of parameters of the first text vectorization model is smaller than the number of parameters of the second text vectorization model, that is, the scale of the first text vectorization model is smaller than the scale of the second text vectorization model, and the first text vectorization model is a randomly initialized model.

[0097] This is to avoid the propagation and expansion of the error of a single model in self-supervision during actual application. That is, since errors are generated in the process of constructing training data based on the second text vectorization model, retraining the second text vectorization model with training data during the training phase will cause the errors to further expand. Therefore, different models are used as the first text vectorization model and the second text vectorization model respectively. At the same time, since the second text vectorization model is only used in the offline stage and is not sensitive to resources and delays, a larger model is used as the second text vectorization model and a relatively smaller model is used as the first text vectorization model.

[0098] In some embodiments, see Figure 3B , Figure 3A Clustering the answer texts in the multiple groups of question and answer texts in step 101 to obtain multiple answer text clusters can be achieved by following the steps 1011 to 1012, which are described in detail below.

[0099] In step 1011, a second text vectorization model trained in advance is used to perform second text vectorization processing on each answer text to obtain multiple answer text vectors.

[0100] In some embodiments, a second text vectorization process is performed on each answer text through a pre-trained second text vectorization model (e.g., Bidirectional Encoder Representations from Transformers (BERT), general semantic vector model (BGE), etc.) to obtain multiple answer text vectors.

[0101] For example, performing a second text vectorization process on each answer text through BERT to obtain the answer text vector of each answer text can be achieved in the following ways: performing word segmentation on the answer text to obtain multiple data units of the answer text; performing embedding process on each data unit to obtain the embedding feature of each data unit; performing position encoding process on each data unit to obtain the position encoding feature of each data unit; fusing the embedding feature and position encoding feature of each data unit to obtain the fused feature of each data unit; performing feature encoding process on the fused feature of each data unit to obtain the answer text vector of the answer text.

[0102] For example, punctuation marks such as spaces, periods, and commas can be used as tokenization marks to perform tokenization processing on the answer text, that is, to divide the answer text into multiple data units.

[0103] For example, the answer text can also be segmented by a tokenizer based on Byte-Pair Encoding (BPE), a tokenizer based on Byte-Level Byte Pair Encoding (BLBPE), etc. The embodiments of the present application do not limit the specific tokenizer (Tokenizer) or tokenization method used to segment the answer text.

[0104] For example, each data unit (such as a word) is embedded to obtain the embedding features of each data unit. This can be achieved in the following ways: Build a vocabulary that contains all the words or symbols that appear in the text. For each word or symbol in the vocabulary, convert it into a vector of a fixed size. This process is called word embeddings. The dimension of word embeddings is a hyperparameter, and you can choose values ​​such as 50, 100, 300, etc. The higher the embedding dimension, the richer the information that the model can capture, but the higher the computational cost. You can learn the embedding matrix of words or symbols through training models such as Word2Vec, GloVe, BERT, etc., so as to convert each data unit into the corresponding embedding feature.

[0105] For example, position encoding (Position Embeddings) is performed on each data unit to obtain the position encoding features of each data unit. The position encoding process can be generated by a fixed algorithm, for example, it can be implemented using a combination of sine and cosine functions. For the dimension index i of the position encoding feature of each position (the position of each data unit in the answer text) (for example, the number of dimensions of the position encoding feature of each data unit is d, then the range value of the dimension index i is from 0 to (d-1)), the position encoding of the dimension with i being an even number uses the sine function, and the position encoding of the dimension with i being an odd number uses the cosine function. The embodiments of the present application do not limit the specific implementation method of the position encoding process.

[0106] Here, position encoding is performed because the sequential information in the answer text will be lost when the answer text is converted into a numerical representation (embedded feature) that can be processed by the second text vectorization model. The purpose of position encoding is to encode the sequential information of the data units so that the second text vectorization model can understand the positional relationship of the data units in the answer text.

[0107] For example, the embedding feature of each data unit is added to the position encoding feature to obtain the first fused feature of each data unit.

[0108] For example, the fused features of each data unit are subjected to attention encoding processing (such as self-attention mechanism, multi-head attention mechanism, etc.) to obtain the attention encoding features of each data unit; the attention encoding features of each data unit are combined to obtain the answer text vector of the answer text.

[0109] For example, before attention encoding, the fused features of each data unit can also be normalized (such as layer normalization) to obtain the normalized features of each data unit. For the normalized features of each data unit, the following attention encoding process is performed: First, a linear transformation is performed to generate three matrices: query vector (Query, Q), key vector (Key, K) and value vector (Value, V). The linear transformation is composed of a learnable weight matrix , and Implementation, next, calculate the dot product of the query vector and the key vector to obtain the attention score (here, the dot product operation is performed on the query vector of the currently processed data unit and the key vectors of all data units (including the currently processed data unit) to obtain the attention scores corresponding to all data units). Next, normalize the attention score, for example, apply a normalization function (such as a softmax function) to convert the attention score into a probability distribution, use the attention probability distribution to weight the value vector, and obtain a rich representation of the correlation of data units at different positions, that is, the attention encoding features of the data unit. Concatenate the attention encoding features of each data unit to obtain the answer text vector of the answer text.

[0110] Through the attention encoding process, a weight (attention probability distribution) can be assigned to each data unit in the sequence, so that the second text vectorization model can better capture long-distance dependencies. By calculating the dot product between the query vector and the key vector, the second text vectorization model can dynamically decide which data units are more important for the representation of the current data unit, thereby improving the sensitivity of the second text vectorization model to the context. By weighted summing the value vector, attention encoding is able to generate a rich representation of the correlation of data units at different positions in the sequence.

[0111] Among them, the encoder for feature encoding processing can be constructed based on a transformer (Transformer). For example, the fused features of each data unit can be feature encoded through the encoder structure (Transformer Encoder) of the transformer to obtain the answer text vector of the answer text. Here, the encoder structure of the transformer can be stacked with multiple layers for encoding processing. The embodiments of the present application do not limit the specific encoder structure.

[0112] In step 1012, multiple answer text vectors are clustered to obtain multiple answer text clusters.

[0113] In some embodiments, a plurality of answer text vectors are clustered by a pre-set clustering algorithm (e.g., K-means algorithm, hierarchical clustering, density-based spatial clustering of applications with noise (DBSCAN), Gaussian mixture model (GMM), etc.), to obtain a plurality of answer text clusters (corresponding to Figure 7 In the answer text cluster 1, answer text cluster 2 and answer text cluster 3), the multiple answer text clusters here can be at least two answer text clusters.

[0114] For example, Figure 7 Answer text in Perform clustering to obtain k clusters (answer text clusters) ,in , that is, m answer texts are placed in the same cluster middle.

[0115] For example, clustering multiple answer text vectors using the K-means algorithm to obtain multiple answer text clusters (for example, at least two answer text clusters) can be achieved in the following way: first, determine the parameters of the K-means algorithm, which include the number of clusters (k value) and other settings of the algorithm (such as initialization method, maximum number of iterations, etc.); next, cluster the answer text vectors using the selected parameters, and clustering adopts the nearest neighbor strategy, that is, the answer text vectors are divided into clusters closest to the cluster center, thereby obtaining multiple answer text clusters, wherein the distance between the answer text vectors can be measured by methods such as Euclidean distance, Manhattan distance, and cosine similarity.

[0116] In some embodiments, see Figure 3C After clustering the answer texts in multiple groups of question and answer texts to obtain multiple answer text clusters, the following steps 1013 to 1014 may be performed, which are described in detail below.

[0117] In step 1013, the intra-cluster similarity of each answer text cluster is determined, wherein the intra-cluster similarity represents the similarity between answer texts in the answer text cluster.

[0118] In some embodiments, the intra-cluster similarity of each answer text cluster can be evaluated by internal indicators (such as Silhouette Coefficient, Calinski-Harabasz Index, etc.). For example, the value range of Silhouette Coefficient or Calinski-Harabasz Index is [-1, 1]. The larger the value, the better the clustering effect (i.e., the higher the intra-cluster similarity).

[0119] In step 1014, answer text clusters whose intra-cluster similarity is less than or equal to an intra-cluster similarity threshold are filtered out from the plurality of question text clusters.

[0120] In some embodiments, answer text clusters whose intra-cluster similarity is less than or equal to an intra-cluster similarity threshold (eg, 0.5) are screened out to obtain remaining answer text clusters.

[0121] In some embodiments, see Figure 3D , Figure 3AThe step 101 of constructing a question text cluster corresponding to each answer text cluster can be implemented by executing the following steps 1015 to 1016 for each answer text cluster, which are described in detail below.

[0122] In step 1015, multiple question texts corresponding to each answer text in the answer text cluster are searched from multiple groups of question and answer texts.

[0123] In some embodiments, based on the one-to-one correspondence between answer texts and question texts, multiple question texts corresponding to each answer text in the answer text cluster are queried from multiple groups of question and answer texts.

[0124] In step 1016, the multiple question texts retrieved are combined into question text clusters corresponding to the answer text clusters.

[0125] In some embodiments, multiple question texts corresponding to each answer text in each answer text cluster are combined into one question text cluster.

[0126] For example, Figure 7 The answer text cluster 1 in is represented as ,Will Each answer text in the multiple question texts corresponding to each answer text is combined into a question text cluster, corresponding to Figure 7 The question text cluster 1 in the question text cluster 1 can be expressed as , where The corresponding question text is .

[0127] In some embodiments, after constructing a question text cluster corresponding to each answer text cluster, before obtaining at least one question positive sample different from the first question text from the first question text cluster, the following processing can also be performed: obtaining the number of texts in each question text cluster; based on the question text clusters whose text number is greater than or equal to the text number threshold, proceeding to obtain at least one question positive sample different from the first question text from the first question text cluster where the first question text is located.

[0128] For example, after constructing the question text cluster corresponding to each answer text cluster, the question text clusters are screened by a preset threshold of the number of texts in the cluster (for example, the number of texts in each question text cluster is not less than 6), and question text clusters whose number of texts in the cluster is greater than or equal to the threshold of the number of texts in the cluster are retained, thereby ensuring that each question text cluster can provide a sufficient number of positive question samples.

[0129] In some embodiments, see Figure 3E , when multiple sets of question-answer texts correspond to different fields, Figure 3AClustering the answer texts in the multiple groups of question and answer texts in step 101 to obtain multiple answer text clusters can be achieved by following the steps 201 to 202, which are described in detail below.

[0130] In step 201, multiple groups of question-answer texts are divided into multiple sets, where each set corresponds to a field.

[0131] In some embodiments, when each group of question and answer texts has a pre-labeled domain label, the multiple groups of question and answer texts are divided into multiple sets according to the domain labels, and each set corresponds to a domain (that is, the question and answer texts in a set correspond to the same domain label).

[0132] For example, the field can refer to the following aspects:

[0133] Industry or professional field: refers to specific industries such as medical, gaming, education, law, engineering, etc. Each industry has its own specific data types, terminology, rules, and focus. For example, data in the gaming field may include game rule data, game technical data (such as game engine configuration, performance data, etc.), game player question text, game customer service reply text, etc.

[0134] Data type or format area: refers to the type of data itself, such as structured data, unstructured data (such as text, pictures, audio), time series data, etc. Different data types can be processed and analyzed in different ways.

[0135] Application field: refers to the specific scenario or purpose in which the data is applied, such as data analysis, machine learning model training, decision support system, etc. For example, in the application field of Natural Language Processing (NLP), data will be used to train language models and perform text analysis.

[0136] Knowledge domain: refers to the scope of knowledge covered by the data, such as physics, biology, psychology, etc. Data sets in these fields include expertise in specific disciplines.

[0137] In some embodiments, when there is no pre-annotated domain label for each group of question and answer texts, the domain label of the domain to which each group of question and answer texts belongs can be identified by using domain-specific keywords, topic models (such as Linear Discriminant Analysis (LDA)), etc.), or other methods, thereby dividing multiple groups of question and answer texts into different sets based on the identified domain labels, each set containing question and answer texts belonging to the same domain.

[0138] In step 202, for each set, the answer texts in the set are clustered to obtain a plurality of answer text clusters in the field corresponding to the set.

[0139] In some embodiments, multiple answer text vectors in a set are clustered using a pre-set clustering algorithm (e.g., K-means algorithm, hierarchical clustering, density-based noise-resistant clustering method, Gaussian mixture model, etc.) to obtain multiple answer text clusters corresponding to the field of the set.

[0140] Continue to see Figure 3A In step 102, at least one question positive sample different from the first question text is obtained from the first question text cluster where the first question text is located, wherein the first question text is any question text.

[0141] For example, the first question text is represented as , indicating the Question text clusters ( ) Question text, Represents the first question text cluster.

[0142] In some embodiments, at least one question text different from the first question text is obtained from the first question text cluster as a positive question sample, wherein the positive question sample and the first question text meet a similarity condition.

[0143] Here, the similarity condition satisfied by the positive sample of the question and the first question text may be that the positive sample of the question and the first question text come from the same question text cluster. The similarity condition may also be that the positive sample of the question and the first question text come from the same question text cluster, and the similarity (e.g., cosine similarity) between the positive sample of the question and the first question text needs to be greater than or equal to a preset threshold (e.g., 0.65). It should be noted that the preset threshold here is different from the first similarity threshold and the second similarity threshold below.

[0144] For example, from Different from At least one positive sample of the problem can be expressed by formula (1):

[0145] (1)

[0146] in, represents the positive sample of the problem, represents random sampling (corresponding to Figure 7 ), Represents the number of positive samples of the problem obtained by sampling, that is, .

[0147] In step 103, at least one negative question sample is obtained from a second question text cluster, wherein the second question text cluster is any question text cluster different from the first question text cluster.

[0148] In some embodiments, a plurality of second question texts are obtained from the second question text cluster, and the plurality of second question texts obtained are screened, and the second question texts remaining after the screening are used as negative question samples, wherein the negative question samples do not meet the similarity condition with the first question text in the first question text cluster.

[0149] In some embodiments, the obtained multiple second question texts are screened and the remaining second question texts after the screening are used as negative question samples. This can be achieved by performing the following processing on each second question text: obtaining a first similarity between the second question text and the first question text, and obtaining a second similarity between the second answer text and the first answer text, wherein the first answer text is the answer text corresponding to the first question text, and the second answer text is the answer text corresponding to the second question text; in response to the first similarity being less than or equal to a first similarity threshold, and the second similarity being less than or equal to a second similarity threshold, the second question text is used as a negative question sample.

[0150] For example, suppose that for the first question text (correspond Figure 7 For the question text in the i-th question text cluster ), except for the first question text The first question text cluster In addition, other question text clusters include A question text, for example, is represented as: (That is, there are M second question texts in total), each second question text and the corresponding second answer text are represented as: , the first question text and the first answer text, as well as each second question text and each second answer text corresponding to the second question text are respectively subjected to text vectorization processing by a pre-trained text vectorization model (such as a general semantic vector model (BGE) etc.) (the specific implementation method can refer to the description of step 1011 above), and the feature vectors of the first question text and the first answer text are obtained. and (here, The feature vector representing the first question text, The feature vector representing the first answer text), and the feature vector of each second question text and the second answer text corresponding to each second question text, are expressed as: The obtained multiple second question texts can be screened by formula (2) (corresponding to Figure 7Inter-cluster filtering in (

[0151] (2)

[0152] in, Indicates the calculation of the similarity between two feature vectors (such as cosine similarity), represents the first similarity, represents the second similarity, represents the first similarity threshold, represents the second similarity threshold, "1" means retaining the second question text as a negative sample, and "0" means filtering out the second question text. Formula (2) ensures that the negative sample does not contain the same question text as the first question text. Similar samples ensure the stability of the contrastive learning of the first text vectorization model.

[0153] For example, random sampling can be performed from the filtered second question text to obtain at least one negative sample of the question, for example, represented by formula (3) (corresponding to Figure 7 Random sampling after inter-cluster filtering in ( ):

[0154] (3)

[0155] in, represents the problem negative sample, represents random sampling, represents the set of the second question texts after screening (filtering) by formula (2), Represents the number of negative samples of the problem obtained by random sampling, that is, .

[0156] In step 104, a first text vectorization model is trained based on the plurality of first question texts and the question positive samples and the question negative samples respectively corresponding to the plurality of first question texts to obtain a trained first text vectorization model.

[0157] For example, for each first question text , Corresponding and Construct a training sample (corresponding to Figure 7 training samples in), for example, represented as: , based on a training data set D consisting of multiple training samples, the first text vectorization model is trained (corresponding to Figure 7 Contrastive learning in ), thereby obtaining the first text vectorization model after training.

[0158] Since the training data (training samples ) does not require manual intervention and can be fully automated, achieving the beneficial effects of reducing model training costs and improving model training efficiency, and can be quickly applied to target business scenarios.

[0159] In some embodiments, see Figure 3F , Figure 3A The step 104 shown can be implemented by following the steps 1041 to 1043, which are described in detail below.

[0160] In step 1041, first text vectorization processing is performed on multiple first question texts, and the problem positive samples and problem negative samples corresponding to the multiple first question texts respectively through a first text vectorization model to obtain a first question text vector for each first question text, a positive sample vector for each problem positive sample, and a negative sample vector for each problem negative sample.

[0161] In some embodiments, through a pre-trained (or initialized) first text vectorization model (such as BERT, BGE, etc.), first text vectorization processing is performed on multiple first question texts, and the problem positive samples and problem negative samples corresponding to the multiple first question texts, respectively, to obtain the first question text vector of each first question text, the positive sample vector of each problem positive sample, and the negative sample vector of each problem negative sample.

[0162] Here, the specific implementation of the first text vectorization processing can refer to the description of the second text vectorization processing in step 1011 above, which will not be repeated here.

[0163] In step 1042, a first loss value is determined based on the first question text vector, the positive sample vector, and the negative sample vector by using a preset loss function.

[0164] In some embodiments, a first loss value is determined based on the first question text vector, the positive sample vector, and the negative sample vector by using a pre-set loss function (comparative learning loss function), which can be expressed by formula (4):

[0165] (4)

[0166] in, represents the first loss value, Indicates the first question text The first question text vector, Indicates the first question text Corresponding Positive sample of questions Middle The positive sample vector of the positive sample of the question, First question text Corresponding Negative examples of questions Middle Negative sample vector of negative samples of the problem, Indicates the calculation of the similarity between two feature vectors (such as cosine similarity), Represents a regulation constant.

[0167] In step 1043, the network parameters of the first text vectorization model are updated based on the first loss value to obtain a trained first text vectorization model.

[0168] In some embodiments, gradient information is obtained through the first loss value, and the parameters of the initialized first text vectorization model are updated according to the gradient information to obtain the trained first text vectorization model.

[0169] For example, the gradient information of the first loss value for each parameter of the first text vectorization model is obtained through the back-propagation algorithm, and the parameters of the initialized first text vectorization model are updated using the obtained gradient information according to the gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations is reached or the first text vectorization model converges, thereby obtaining the trained first text vectorization model.

[0170] Through steps 101 to 104, the answer text corresponding to each question text is used as an anchor point to cluster the answer texts. After obtaining the answer text clusters, the question text clusters corresponding to each answer text cluster are constructed according to the clustering results of the answer texts. This makes full use of the correlation between the question texts and the answer texts, avoids the problem of poor effect of directly clustering the question texts due to the diversity and dispersion of the question text expressions, improves the clustering effect of the question texts, and thus improves the quality of the question positive samples and the question negative samples obtained through the intra-cluster and inter-cluster relationships of the question texts, and finally improves the text representation ability of the first text vectorization model trained based on the first question text and the question positive samples and the question negative samples corresponding to the first question text.

[0171] In some embodiments, when multiple groups of question and answer texts correspond to different fields, for each set, based on multiple first question texts in the set, and question positive samples and question negative samples corresponding to the multiple first question texts respectively, a first text vectorization model of the field corresponding to the set is trained to obtain a trained first text vectorization model of the field corresponding to the set.

[0172] In some embodiments, when multiple groups of question and answer texts correspond to a first field, the first text vectorization model is fine-tuned based on multiple groups of question and answer texts in a second field to obtain a first text vectorization model for the second field, wherein the first field is different from the second field.

[0173] In some embodiments, see Figure 3G , based on multiple groups of question and answer texts in the second field, the first text vectorization model is fine-tuned to obtain the first text vectorization model of the second field, which can be achieved by following steps 301 to 305, which are described in detail below.

[0174] In step 301, first text vectorization processing is performed on multiple first question texts in the second field and problem positive samples and problem negative samples corresponding to multiple first question texts in the second field through a first text vectorization model to obtain first question text vectors of the first question texts in the second field, positive sample vectors of the problem positive samples in the second field, and negative sample vectors of the problem negative samples in the second field.

[0175] In some embodiments, through a pre-trained (or initialized) first text vectorization model (such as BERT, BGE, etc.), first text vectorization processing is performed on multiple first question texts in the second field and problem positive samples and problem negative samples corresponding to multiple first question texts in the second field, respectively, to obtain first question text vectors of the first question texts in the second field, positive sample vectors of each problem positive sample in the second field, and negative sample vectors of each problem negative sample.

[0176] Here, the specific implementation of the first text vectorization processing can refer to the description of the second text vectorization processing in step 1011 above, which will not be repeated here.

[0177] In step 302, a second loss value is determined based on the first question text vector of the second domain, the positive sample vector of the second domain, and the negative sample vector of the second domain through a preset loss function.

[0178] In some embodiments, a second loss value is determined based on the first question text vector of the second domain, the positive sample vector of the second domain, and the negative sample vector of the second domain through a pre-set loss function (comparative learning loss function), as described in formula (4) above.

[0179] In step 303, a first low-rank matrix initialized based on Gaussian distribution is generated, and a second low-rank matrix initialized based on all zeros is generated.

[0180] In some embodiments, a first low-rank matrix is ​​initialized using random numbers generated using a Gaussian distribution (normal distribution), where the Gaussian distribution is defined by mean and standard deviation (STD) parameters, and all elements of a second low-rank matrix are initialized to zero, wherein the dimension of the second low-rank matrix is ​​the same as the transpose of the first low-rank matrix.

[0181] In step 304, fine-tuning weight parameters are constructed based on the first low-rank matrix and the second low-rank matrix, and the fine-tuning weight parameters are added to the network parameters of the first text vectorization model.

[0182] In some embodiments, the product of the first low-rank matrix (A) and the second low-rank matrix (B) is used as the fine-tuning weight parameter (∆W), wherein the dimension of the second low-rank matrix is ​​the same as the transpose of the first low-rank matrix.

[0183] For example, the fine-tuning weight parameter is expressed as ΔW=B×A, where B corresponds to the second low-rank matrix, A corresponds to the first low-rank matrix, and the original parameter (weight parameter) of the first text vectorization model is expressed as W. It is fine-tuned by a fine-tuning weight parameter (ΔW), so that the weight parameter of the fine-tuned (i.e. updated) first text vectorization model is W+ΔW.

[0184] In step 305, the weight parameters are updated and fine-tuned using the second loss value to obtain a first text vectorization model for the second domain.

[0185] In some embodiments, the original parameters of the first text vectorization model except the first low-rank matrix and the second low-rank matrix are kept unchanged, and the first low-rank matrix and the second low-rank matrix are updated by the second loss value.

[0186] Continuing with the above example, in the reverse update process, W is frozen (W is not updated, corresponding to keeping the parameters of the network except the first low-rank matrix and the second low-rank matrix unchanged), and only the parameters of A and B (i.e., ΔW) are updated, thereby obtaining the first text vectorization model of the second field.

[0187] In other embodiments, see Figure 3H , clustering the answer texts in multiple groups of question and answer texts to obtain multiple answer text clusters is achieved through the second text vectorization model. Before training the first text vectorization model, the following steps 401 to 403 can also be performed, which are described in detail below.

[0188] In step 401, importance evaluation is performed on network parameters of each network layer of the second text vectorization model to obtain importance parameters of each network layer.

[0189] In some embodiments, the absolute value norm (L1-norm) of the network parameters (weight matrix) of each network layer is obtained as the importance parameter of each network layer.

[0190] Here, the L1 norm is used as the importance parameter of each network layer because the L1 norm tends to produce a sparse weight matrix, which means that there will be more weights close to zero in the network. In a sparse matrix, the position of the non-zero element represents the most important connection, so the L1 norm can emphasize those layers that contribute the most to the network output. Therefore, the L1 norm can reflect the contribution of each layer to the network output, thereby guiding the optimization and simplification of the model.

[0191] In step 402, according to the preset pruning ratio and importance parameters, the network layer to be pruned is determined, and the network layer to be pruned is removed from the second text vectorization model to obtain the pruned second text vectorization model.

[0192] In some embodiments, according to a preset pruning ratio, for example, 20%, the network layers whose importance parameters are ranked in the bottom 20% are used as the network layers to be pruned, and the network parameters of the network layers to be pruned are removed to obtain a pruned second text vectorization model.

[0193] For example, first, before starting pruning, set the pruning ratio (for example, 20%). This ratio refers to the ratio of the network layer with the smallest weight that will be removed after sorting by importance in the entire network. Next, calculate the importance parameter of each network layer (such as L1 norm). This parameter reflects the importance of the layer in the network, that is, the contribution to the overall model performance. Further, all network layers are sorted according to their importance parameters, and then according to the pre-set pruning ratio, the network layer with the lowest ranking and the lowest importance is selected as the layer to be pruned. Finally, for the selected network layer to be pruned, set the weight parameter to zero or completely remove the layer from the model to obtain the pruned second text vectorization model. You can also verify the pruned model to ensure that the performance of the model is not greatly affected. If the performance drops too much, you can adjust the pruning ratio.

[0194] For example, the pruned second text vectorization model can also be fine-tuned to recover the performance loss caused by pruning. For example, based on multiple groups of question and answer texts, the pruned second text vectorization model is fine-tuned. The specific implementation of the fine-tuning can refer to the description of steps 301 to 305 above, which will not be repeated here.

[0195] In step 403, the pruned second text vectorization model is used as the first text vectorization model to be trained.

[0196] For example, the pruned second text vectorization model is used as the first text vectorization model to be trained, and the first text vectorization model to be trained is trained based on multiple first question texts, and problem positive samples and problem negative samples corresponding to the multiple first question texts, to obtain the trained first text vectorization model.

[0197] The following will combine the exemplary application and implementation of the server provided in the embodiment of the present application, taking the server as the execution subject, to illustrate the text processing method provided in the embodiment of the present application. Figure 4 , Figure 4 is a flowchart of a text processing method provided in an embodiment of the present application, which will be combined with Figure 4 The steps shown are explained.

[0198] In step 501, a plurality of object question texts are obtained.

[0199] In some embodiments, a plurality of object question texts (user question texts) are obtained from a database.

[0200] For example, the questions input by the user in the human-computer interaction interface can be captured and stored in a backend database, or the user's question text can be extracted by parsing the content of the user's email and stored in the database.

[0201] In step 502, a plurality of object question texts are vectorized by a first text vectorization model to obtain a plurality of object question text vectors, wherein the first text vectorization model is trained by a text vectorization model training method provided in an embodiment of the present application.

[0202] In some embodiments, the following processing is performed on each object question text through a first text vectorization model (the specific training process can be found in the description of steps 101 to 104 above): the object question text is segmented to obtain multiple data units of the object question text; each data unit is embedded to obtain an embedding feature of each data unit; each data unit is position encoded to obtain a position encoding feature of each data unit; the embedding feature and the position encoding feature of each data unit are fused to obtain a fused feature of each data unit; the fused feature of each data unit is feature encoded to obtain an object question text vector of the object question text.

[0203] For example, punctuation marks such as spaces, periods, and commas may be used as segmentation marks to perform segmentation processing on the object question text, that is, to divide the object question text into multiple data units (eg, multiple words).

[0204] For example, the object question text can also be segmented by a word segmenter based on byte pair encoding (BPE), a word segmenter based on byte-level byte pair encoding (BLBPE), etc. The embodiments of the present application do not limit the specific word segmenter or word segmentation method used to segment the object question text.

[0205] For example, each data unit is embedded to obtain the embedding features of each data unit. This can be achieved in the following way: build a vocabulary, which contains all the words or symbols that appear in the text. For each word or symbol in the vocabulary, convert it into a vector of a fixed size. This process is called word embedding. The dimension of word embedding is a hyperparameter, and you can choose values ​​such as 50, 100, 300, etc. The higher the embedding dimension, the richer the information that the model can capture, but the higher the computational cost. Each data unit can be converted into the corresponding embedding feature through the trained word embedding matrix.

[0206] For example, position coding is performed on each data unit to obtain a position coding feature of each data unit. The position coding process can be generated by a fixed algorithm, for example, it can be implemented using a combination of sine and cosine functions. For the dimension index i of the position coding feature of each position (the position of each data unit in the object question text) (for example, the number of dimensions of the position coding feature of each data unit is d, then the range value of the dimension index i is from 0 to (d-1)), the position coding of the dimension with i being an even number uses a sine function, and the position coding of the dimension with i being an odd number uses a cosine function. The embodiments of the present application do not limit the specific implementation method of the position coding process.

[0207] For example, the embedding feature of each data unit is added to the position encoding feature to obtain the first fused feature of each data unit.

[0208] For example, the fused features of each data unit are subjected to attention encoding processing (such as self-attention mechanism, multi-head attention mechanism, etc.) to obtain the attention encoding features of each data unit; the attention encoding features of each data unit are combined to obtain the object question text vector of the object question text.

[0209] For example, before attention encoding, the fused features of each data unit can also be normalized (such as layer normalization) to obtain the normalized features of each data unit. For the normalized features of each data unit, the following attention encoding process is performed: First, a linear transformation is performed to generate three matrices: query vector (Q), key vector (K) and value vector (V). The linear transformation is performed by the trained weight matrix , and Implementation, next, calculate the dot product of the query vector and the key vector to obtain the attention score (here, the dot product operation is performed on the query vector of the currently processed word and the key vectors of all words (including the currently processed word) to obtain the attention scores corresponding to all words). Next, normalize the attention score, for example, apply a normalization function (such as a softmax function) to convert the attention score into a probability distribution, and use the attention probability distribution to weight the value vector to obtain a rich representation of the relevance of words in different positions, that is, the attention encoding features of the words, and concatenate the attention encoding features of the words into the object question text vector of the object question text.

[0210] In step 503, multiple object question text vectors are clustered to obtain at least one object question text cluster.

[0211] In some embodiments, multiple object question text vectors are clustered using a pre-set clustering algorithm (e.g., K-means algorithm, hierarchical clustering, density-based noise-resistant clustering method, Gaussian mixture model, etc.) to obtain at least one object question text cluster. The embodiments of the present application do not limit the specific clustering algorithm.

[0212] For example, clustering multiple object question text vectors using the K-means algorithm to obtain at least one object question text cluster can be achieved in the following way: first, determine the parameters of the K-means algorithm, the parameters include the number of clusters (k value) and other settings of the algorithm (such as initialization method, maximum number of iterations, etc.), then use the selected parameters to cluster the object question text vectors, and clustering adopts the nearest neighbor strategy, that is, the object question text vectors are divided into clusters closest to the cluster center, thereby obtaining at least one object question text cluster, wherein the distance between the object question text vectors can be measured by methods such as Euclidean distance, Manhattan distance, and cosine similarity.

[0213] In step 504, the answer pre-associated with each object question text cluster is obtained as the answer in response to the object question text in each object question text cluster.

[0214] In some embodiments, by querying the mapping table, the answer (or response template text) pre-associated with each object question text cluster is obtained as the answer to the object question text in each object question text cluster.

[0215] For example, descriptive text extraction is performed on each object question text cluster to obtain a descriptive text label for each object question text cluster; based on the mapping relationship (one-to-one correspondence) between the descriptive text label and the answer in the mapping table, the answer corresponding to the descriptive text label is obtained as the answer to the object question text in the object question text cluster.

[0216] For example, descriptive text extraction is performed on each object question text cluster to obtain a descriptive text label for each object question text cluster. This can be achieved in any of the following ways: obtaining the object question text closest to the center of the object question text cluster as the descriptive text label; obtaining at least one keyword in the object question text cluster as the descriptive text label; generating a descriptive text (or summary text) of the object question text cluster as the descriptive text label through a pre-trained summary sentence generation model.

[0217] For example, the distance between the object question texts in the object question text cluster can be measured by methods such as Euclidean distance, Manhattan distance, and cosine similarity.

[0218] For example, obtaining at least one keyword in the object question text cluster as a descriptive text label can be achieved in the following ways: preprocessing each object question text, such as removing stop words, punctuation marks, special characters, etc., using a keyword extraction algorithm (such as the term frequency-inverse document frequency statistical method (TF-IDF), etc.) to extract keywords from each object question text. These keywords are usually words that tend to express the theme or main content of the object question text, performing frequency statistics on the extracted keywords, and calculating the number of occurrences or frequency of each keyword in the object question text cluster. For example, using a counter object in Python or other statistical methods to implement keyword frequency statistics, thereby obtaining multiple keyword frequency values, and using the keywords with the top k keyword frequency values ​​as descriptive text labels.

[0219] For example, a summary sentence generation model is used to generate description text (or summary text) of an object question text cluster as a description text label. This can be achieved in the following way: a summary sentence generation model (such as BERT) is used to input all object question texts in an object question text cluster through a text input layer, and then processed through word embeddings, interval segment embeddings, position embeddings, and summarization layers to generate description text of the object question text cluster as a description text label.

[0220] In other embodiments, a reply text (answer) is manually set for each object question text cluster, that is, for the object question texts in the same object question text cluster, batch responses are performed through manually input reply texts.

[0221] It should be noted that the embodiments of the present application can be applied to various scenarios that require text processing, such as intelligent customer service scenarios, text retrieval scenarios, text recommendation scenarios, etc. In the field of games, the embodiments of the present application can be applied to game customer service, game recommendation systems, game content review, etc., as illustrated below.

[0222] 1) Game customer service scenario:

[0223] Intelligent answering: The text vectorization model can be used in game customer service robots to analyze players' questions and quickly match answer templates in the database to provide instant customer service support. Alternatively, the text of questions from multiple players can be vectorized through the vectorization model, and the vectorization results can be clustered to provide unified responses to user questions in the same cluster, thereby improving the efficiency of answering.

[0224] Sentiment analysis: Using text vectorization models to perform sentiment analysis can help the customer service team understand players’ emotions and needs, thereby providing more considerate services.

[0225] 2) Game recommendation system. For example, text vectorization models can vectorize the features of game content (such as game introduction, etc.) and player preferences (such as the game types that players like, etc.), thereby recommending games or game content that players may be interested in.

[0226] 3) Game content review. For example, text vectorization models can assist the automatic review system by analyzing the text content such as dialogues and comments in the game to identify and filter inappropriate or sensitive content. For example, if a comment in the game is marked as negative, the comments marked as negative can be further reviewed manually or directly filtered out (such as hiding related comments in the comment interface).

[0227] Next, an exemplary application of the present application embodiment in a game customer service scenario will be described. Figure 5 , Figure 5 is a schematic diagram of the application flow of the text processing method provided in the embodiment of the present application, which will be combined with Figure 5 The steps shown are explained.

[0228] In step 601, multiple user question texts are obtained.

[0229] In some embodiments, multiple user question texts (game players' question texts) correspond to different users, and the multiple user question texts can be obtained in the following manner: time-window-based sampling is performed on the clients (e.g., instant messaging clients, game clients, etc.) of multiple users (e.g., more than 3 users) to obtain user question texts corresponding to the multiple users in the current time window.

[0230] For example, a software development kit (SDK) is integrated in each client application. The user's user question text is collected through the SDK. The length of the sampling time window (for example, 1 minute) and the step length (for example, 30 seconds) can be set by configuring the SDK. The SDK detects the user's user question text and records the timestamp of each user question text, and stores the user question text locally until the sampling time window ends. At the end of the time window, the SDK packages the collected operation data into a data packet and sends it to the server. For example, the SDK sends the data packet to the server through a Hypertext Transfer Protocol request (HTTP) or other protocols. Asynchronous requests can be used to avoid blocking the operation of the client application. The server sets a corresponding interface to receive the data packet sent by the client, aggregates the received data packets, and stores them in a database (for example, MySQL, HIVE, etc.).

[0231] In step 602, a plurality of user question texts are vectorized by a first text vectorization model to obtain a plurality of user question text vectors, wherein the first text vectorization model is trained by a training method for a text vectorization model provided in an embodiment of the present application.

[0232] In some embodiments, the following processing is performed on each user question text through a first text vectorization model (the specific model training process can be found in the description of steps 101 to 104 above): the user question text is segmented to obtain multiple data units of the user question text; each data unit is embedded to obtain an embedding feature of each data unit; each data unit is position encoded to obtain a position encoding feature of each data unit; the embedding feature and the position encoding feature of each data unit are fused to obtain a fused feature of each data unit; the fused feature of each data unit is feature encoded to obtain a user question text vector of the user question text.

[0233] For example, punctuation marks such as spaces, periods, and commas may be used as segmentation marks to perform segmentation processing on the user question text, that is, to divide the user question text into multiple data units (eg, multiple words).

[0234] For example, the user question text can also be segmented by a word segmenter based on byte pair encoding (BPE), a word segmenter based on byte-level byte pair encoding (BLBPE), etc. The embodiments of the present application do not limit the specific word segmenter or word segmentation method used to segment the user question text.

[0235] For example, each data unit is embedded to obtain the embedding features of each data unit. This can be achieved in the following way: build a vocabulary, which contains all the words or symbols that appear in the text. For each word or symbol in the vocabulary, convert it into a vector of a fixed size. This process is called word embedding. The dimension of word embedding is a hyperparameter, and you can choose values ​​such as 50, 100, 300, etc. The higher the embedding dimension, the richer the information that the model can capture, but the higher the computational cost. Each data unit can be converted into the corresponding embedding feature through the trained word embedding matrix.

[0236] For example, position coding is performed on each data unit to obtain a position coding feature of each data unit. The position coding process can be generated by a fixed algorithm, for example, it can be implemented using a combination of sine and cosine functions. For the dimension index i of the position coding feature of each position (the position of each data unit in the user's question text) (for example, the number of dimensions of the position coding feature of each data unit is d, then the range value of the dimension index i is from 0 to (d-1)), the position coding of the dimension with i being an even number uses a sine function, and the position coding of the dimension with i being an odd number uses a cosine function. The embodiments of the present application do not limit the specific implementation method of the position coding process.

[0237] For example, the embedding feature of each data unit is added to the position encoding feature to obtain the first fused feature of each data unit.

[0238] For example, the fused features of each data unit are subjected to attention encoding processing (such as self-attention mechanism, multi-head attention mechanism, etc.) to obtain the attention encoding features of each data unit; the attention encoding features of each data unit are combined to obtain the user question text vector of the user question text.

[0239] For example, before attention encoding, the fused features of each data unit can also be normalized (such as layer normalization) to obtain the normalized features of each data unit. For the normalized features of each data unit, the following attention encoding process is performed: First, a linear transformation is performed to generate three matrices: query vector (Q), key vector (K) and value vector (V). The linear transformation is performed by the trained weight matrix , and Implementation, next, calculate the dot product of the query vector and the key vector to obtain the attention score (here, the dot product operation is performed on the query vector of the currently processed word and the key vectors of all words (including the currently processed word) to obtain the attention scores corresponding to all words). Next, normalize the attention score, for example, apply a normalization function (such as a softmax function) to convert the attention score into a probability distribution, and use the attention probability distribution to weight the value vector to obtain a rich representation of the relevance of words in different positions, that is, the attention encoding features of the words, and concatenate the attention encoding features of the words into a user question text vector of the user question text.

[0240] In step 603, multiple user question text vectors are clustered to obtain at least one user question text cluster.

[0241] In some embodiments, multiple user question text vectors are clustered using a pre-set clustering algorithm (for example, K-means algorithm, hierarchical clustering, density-based noise-resistant clustering method, Gaussian mixture model, etc.) to obtain at least one user question text cluster. The embodiments of the present application do not limit the specific clustering algorithm.

[0242] In step 604 , the answer pre-associated with each user question text cluster is obtained as an answer in response to the user question text in each user question text cluster.

[0243] In some embodiments, by querying the mapping table, the answer (or response template text) pre-associated with each user question text cluster is obtained as the answer to the user question text in each user question text cluster.

[0244] For example, a description text is extracted for each user question text cluster to obtain a description text label for each user question text cluster (e.g. Figure 8 "Virtual resource exchange", "Customer service phone and status inquiry", "Disagreement with penalty judgment", etc. shown in the text); according to the mapping relationship between the description text label and the answer in the mapping table (one-to-one correspondence), obtain the answer corresponding to the description text label as the answer to the user question text in the user question text cluster.

[0245] For example, descriptive text extraction is performed on each user question text cluster to obtain a descriptive text label for each user question text cluster. This can be achieved in any of the following ways: obtaining the user question text closest to the center of the user question text cluster as the descriptive text label; obtaining at least one keyword in the user question text cluster as the descriptive text label; generating a descriptive text (or summary text) of the user question text cluster as the descriptive text label through a pre-trained summary sentence generation model.

[0246] For example, the distance between user question texts in a user question text cluster can be measured by methods such as Euclidean distance, Manhattan distance, and cosine similarity.

[0247] For example, obtaining at least one keyword in the user question text cluster as a descriptive text label can be achieved in the following ways: preprocessing each user question text, such as removing stop words, punctuation marks, special characters, etc., using a keyword extraction algorithm (such as the term frequency-inverse document frequency statistics method (TF-IDF)), etc.) to extract keywords from each user question text. These keywords are usually words that tend to express the theme or main content of the user question text, performing frequency statistics on the extracted keywords, and calculating the number of occurrences or frequency of each keyword in the user question text cluster. For example, using a counter object in Python or other statistical methods to implement keyword frequency statistics, thereby obtaining multiple keyword frequency values, and using the keywords with the top k keyword frequency values ​​as descriptive text labels.

[0248] For example, a summary sentence generation model is used to generate a description text (or summary text) of a user question text cluster as a description text label. This can be achieved in the following way: a summary sentence generation model (such as BERT) is used to input all user question texts in the user question text cluster through a text input layer, and then processed through word embeddings, interval segment embeddings, position embeddings, and summarization layers to generate a description text of the user question text cluster as a description text label.

[0249] In other embodiments, a reply text (answer) is manually set for each user question text cluster, that is, for user question texts in the same user question text cluster, batch responses are made through manually input reply texts.

[0250] For example, see Figure 8 , multiple description text labels are displayed in the smart reply interface of similar questions (such as Figure 8 Here, a description text label corresponds to a user question text cluster, Figure 8 The figure shows user questions from 6 users including the description text label "virtual resource exchange". Figure 8 In the "Response Strategy" in the , enter the relevant response (or automatically obtain the response text pre-associated with the description text tag), and the customer service will send the same response text to answer user questions in batches. The customer service can click Figure 8 The "Update Similar Sessions" control in the taskbar obtains the user question text data of the next time step from the database, thereby performing a new round of batch user question answering.

[0251] Through steps 601 to 604, the first text vectorization model trained by the training method of the text vectorization model provided in the embodiment of the present application is used to vectorize multiple user questions and cluster the obtained multiple user question text vectors, so that similar user questions can be aggregated together, making it easier for customer service personnel to send a unified reply. In this way, the customer service team can quickly respond to a large number of similar questions and improve the response efficiency. Users can get responses immediately without having to wait for a long time, thereby improving user satisfaction. At the same time, this batch processing method allows the customer service team to focus on more complex and individual user problems, thereby achieving the beneficial effect of saving human resources and reducing operating costs.

[0252] In order to illustrate the effects of the training method and text processing method of the text vectorization model provided in the embodiment of the present application, the general semantic vector model (BGE) is used as the initial first text vectorization model (i.e., the first text vectorization model to be trained). In the same customer service scenario, the performance of the general semantic vector model trained using the initial general semantic vector model and the general semantic vector model trained using the training method of the text vectorization model provided in the embodiment of the present application is compared. Here, the difference between the clustering results obtained by clustering the text using the same clustering algorithm after the text is vectorized by the above two models respectively is compared. The manually labeled clustering accuracy (i.e., the accuracy of the clustering results determined by the manually labeled real labels) and mutual information entropy (when the mutual information entropy is close to 1, it means that the clustering results are very consistent with the real labels and the clustering accuracy is high) are used for evaluation, as shown in Table 1:

[0253] Table 1

[0254]

[0255] It can be seen from Table 1 that the training method and text processing method of the text vectorization model provided in the embodiment of the present application have a certain improvement in the accuracy of clustering. At the same time, the training data (corresponding to the training samples above) in the training method of the text vectorization model provided in the embodiment of the present application ) does not require manual intervention and can be fully automated, achieving the beneficial effects of reducing model training costs and improving model training efficiency, and can be quickly applied to target business scenarios.

[0256] The following is a description of an exemplary structure of a software module implemented as the training device 133 of the text vectorization model provided in the embodiment of the present application. In some embodiments, for example Figure 2A As shown, the software modules stored in the training device 133 of the text vectorization model in the memory 130-1 may include:

[0257] The data processing module 1331 is used to cluster the answer texts in multiple groups of question and answer texts to obtain multiple answer text clusters, and to construct a question text cluster corresponding to each of the answer text clusters, wherein each group of the question and answer texts includes a question text and the answer text corresponding to the question text.

[0258] In some embodiments, the data processing module 1331 is further used to obtain at least one question positive sample different from the first question text from the first question text cluster where the first question text is located, wherein the first question text is any one of the question texts.

[0259] In some embodiments, the data processing module 1331 is further used to obtain at least one negative question sample from a second question text cluster, wherein the second question text cluster is any question text cluster different from the first question text cluster.

[0260] The training module 1332 is used to train a first text vectorization model based on the multiple first question texts, the question positive samples and the question negative samples respectively corresponding to the multiple first question texts, so as to obtain a trained first text vectorization model.

[0261] In some embodiments, the data processing module 1331 is also used to perform the following processing for each of the answer text clusters: querying multiple question texts corresponding to each of the answer texts in the answer text cluster from the multiple groups of question and answer texts; and combining the multiple question texts that have been queried into a question text cluster corresponding to the answer text cluster.

[0262] In some embodiments, the data processing module 1331 is further used to obtain at least one question text different from the first question text from the first question text cluster as a positive question sample, wherein the positive question sample and the first question text meet a similarity condition.

[0263] In some embodiments, the data processing module 1331 is further used to obtain multiple second question texts from the second question text cluster, and to filter the multiple second question texts obtained, and to use the second question texts remaining after the filtering as negative question samples, wherein the negative question samples do not meet the similarity condition with the first question text in the first question text cluster.

[0264] In some embodiments, the data processing module 1331 is also used to perform the following processing on each of the second question texts: obtaining a first similarity between the second question text and the first question text, and obtaining a second similarity between the second answer text and the first answer text, wherein the first answer text is the answer text corresponding to the first question text, and the second answer text is the answer text corresponding to the second question text; in response to the first similarity being less than or equal to a first similarity threshold, and the second similarity being less than or equal to a second similarity threshold, taking the second question text as a negative sample of the question.

[0265] In some embodiments, the training module 1332 is also used to perform first text vectorization processing on multiple first question texts, the problem positive samples and the problem negative samples respectively corresponding to the multiple first question texts through the first text vectorization model to obtain the first question text vector of each first question text, the positive sample vector of each of the problem positive samples and the negative sample vector of each of the problem negative samples; determine a first loss value based on the first question text vector, the positive sample vector and the negative sample vector through a pre-set loss function; update the network parameters of the first text vectorization model based on the first loss value to obtain the trained first text vectorization model.

[0266] In some embodiments, the data processing module 1331 is also used to perform second text vectorization processing on each of the answer texts through a pre-trained second text vectorization model to obtain multiple answer text vectors; and cluster the multiple answer text vectors to obtain multiple answer text clusters.

[0267] In some embodiments, the training module 1332 is also used to evaluate the importance of the network parameters of each network layer of the second text vectorization model to obtain the importance parameters of each network layer; determine the network layer to be pruned according to the preset pruning ratio and the importance parameters, and remove the network layer to be pruned from the second text vectorization model to obtain the pruned second text vectorization model; and use the pruned second text vectorization model as the first text vectorization model to be trained.

[0268] In some embodiments, the data processing module 1331 is also used to determine the intra-cluster similarity of each of the answer text clusters, wherein the intra-cluster similarity represents the similarity between the answer texts in the answer text cluster; and filter out the answer text clusters whose intra-cluster similarity is less than or equal to the intra-cluster similarity threshold from the multiple question text clusters.

[0269] In some embodiments, the data processing module 1331 is also used to obtain the number of texts in each of the question text clusters; based on the question text clusters whose text number is greater than or equal to a text number threshold, proceed to execute the step of obtaining at least one question positive sample that is different from the first question text from the first question text cluster where the first question text is located.

[0270] In some embodiments, when the multiple groups of question and answer texts correspond to different fields, the data processing module 1331 is also used to divide the multiple groups of question and answer texts into multiple sets, wherein each set corresponds to one field; for each set, the answer texts in the set are clustered to obtain multiple answer text clusters of the field corresponding to the set.

[0271] In some embodiments, when the multiple groups of question and answer texts correspond to different fields, the training module 1332 is also used to train, for each of the sets, a first text vectorization model of the field corresponding to the set based on the multiple first question texts in the set, the question positive samples and the question negative samples respectively corresponding to the multiple first question texts, to obtain a trained first text vectorization model of the field corresponding to the set.

[0272] In some embodiments, when the multiple groups of question and answer texts correspond to a first field, the training module 1332 is further used to fine-tune the first text vectorization model based on multiple groups of question and answer texts in a second field to obtain a first text vectorization model for the second field, wherein the first field is different from the second field.

[0273] In some embodiments, the training module 1332 is also used to perform the first text vectorization processing on multiple first question texts in the second field and problem positive samples and problem negative samples corresponding to multiple first question texts in the second field through the first text vectorization model to obtain first question text vectors of the first question texts in the second field, positive sample vectors of problem positive samples in the second field, and negative sample vectors of problem negative samples in the second field; determine a second loss value based on the first question text vectors in the second field, the positive sample vectors in the second field, and the negative sample vectors in the second field through a pre-set loss function; generate a first low-rank matrix initialized based on Gaussian distribution, and generate a second low-rank matrix initialized based on all zeros; construct fine-tuning weight parameters based on the first low-rank matrix and the second low-rank matrix, and add the fine-tuning weight parameters to the network parameters of the first text vectorization model; update the fine-tuning weight parameters through the second loss value to obtain the first text vectorization model of the second field.

[0274] The following is a description of an exemplary structure of the text processing device 134 provided in the present application as a software module. In some embodiments, Figure 2B As shown, the software modules stored in the text processing device 134 of the memory 130-2 may include:

[0275] The data acquisition module 1341 is used to acquire multiple object question texts.

[0276] The data processing module 1342 is used to vectorize the multiple object question texts through a first text vectorization model to obtain multiple object question text vectors, wherein the first text vectorization model is trained by the training method of the text vectorization model provided in an embodiment of the present application.

[0277] In some embodiments, the data processing module 1342 is further used to perform clustering processing on the multiple object question text vectors to obtain at least one object question text cluster.

[0278] In some embodiments, the data processing module 1342 is further configured to obtain an answer pre-associated with each of the object question text clusters as an answer in response to the object question text in each of the object question text clusters.

[0279] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the training method of the text vectorization model or the text processing method described in the embodiment of the present application.

[0280] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the text vectorization model training method or text processing method provided in the present application embodiment, for example, Figure 3A The training method of the text vectorization model shown, or Figure 4 The text processing method shown.

[0281] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0282] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0283] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0284] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0285] To summarize, through the embodiments of the present application, the answer text corresponding to each question text is used as an anchor point to cluster the answer texts. After obtaining the answer text clusters, the question text clusters corresponding to each answer text cluster are constructed based on the clustering results of the answer texts. This makes full use of the correlation between the question texts and the answer texts, avoids the problem of poor effect of directly clustering the question texts due to the diversity and dispersion of the question text expressions, improves the clustering effect of the question texts, and thus improves the quality of the question positive samples and question negative samples obtained through the intra-cluster and inter-cluster relationships of the question texts, and ultimately improves the text representation ability of the first text vectorization model trained based on the first question text and the question positive samples and question negative samples corresponding to the first question text.

[0286] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A training method for a text vectorization model, characterized in that: The method comprises: Clustering the answer texts in the multiple groups of question and answer texts to obtain multiple answer text clusters, and constructing a question text cluster corresponding to each of the answer text clusters, wherein each group of the question and answer texts includes a question text and the answer text corresponding to the question text; Acquire at least one question positive sample different from the first question text from the first question text cluster where the first question text is located, wherein the first question text is any one of the question texts; Acquire at least one question negative sample from a second question text cluster, wherein the second question text cluster is any question text cluster different from the first question text cluster; Based on the plurality of first question texts, the question positive samples and the question negative samples respectively corresponding to the plurality of first question texts, a first text vectorization model is trained to obtain a trained first text vectorization model.

2. The method according to claim 1, characterized in that The step of constructing a question text cluster corresponding to each answer text cluster includes: The following processing is performed for each of the answer text clusters: From the plurality of groups of question-answer texts, query a plurality of question texts corresponding to each of the answer texts in the answer text cluster; The multiple question texts retrieved are combined into a question text cluster corresponding to the answer text cluster.

3. The method according to claim 1, characterized in that The step of obtaining at least one positive question sample different from the first question text from the first question text cluster where the first question text is located includes: At least one question text different from the first question text is obtained from the first question text cluster as a positive question sample, wherein the positive question sample and the first question text meet a similarity condition.

4. The method according to claim 1, characterized in that: The step of obtaining at least one negative question sample from the second question text cluster includes: A plurality of second question texts are obtained from the second question text cluster, and the plurality of second question texts are screened, and the second question texts remaining after the screening are used as negative question samples, wherein the negative question sample does not satisfy a similarity condition with the first question text in the first question text cluster.

5. The method according to claim 4, characterized in that The step of screening the obtained plurality of second question texts and taking the second question texts remaining after screening as question negative samples includes: The following processing is performed on each of the second question texts: Obtaining a first similarity between a second question text and a first question text, and obtaining a second similarity between a second answer text and the first answer text, wherein the first answer text is an answer text corresponding to the first question text, and the second answer text is an answer text corresponding to the second question text; In response to the first similarity being less than or equal to a first similarity threshold, and the second similarity being less than or equal to a second similarity threshold, the second question text is used as a negative question sample.

6. The method according to any one of claims 1 to 5, characterized in that: The step of training a first text vectorization model based on the plurality of first question texts, the question positive samples and the question negative samples respectively corresponding to the plurality of first question texts, and obtaining a trained first text vectorization model includes: Performing first text vectorization processing on the plurality of first question texts, the question positive samples and the question negative samples respectively corresponding to the plurality of first question texts through the first text vectorization model, to obtain a first question text vector of each of the first question texts, a positive sample vector of each of the question positive samples and a negative sample vector of each of the question negative samples; Determine a first loss value based on the first question text vector, the positive sample vector, and the negative sample vector by using a preset loss function; The network parameters of the first text vectorization model are updated based on the first loss value to obtain a trained first text vectorization model.

7. The method according to any one of claims 1 to 5, characterized in that: The answer texts in the multiple groups of question and answer texts are clustered to obtain multiple answer text clusters, including: Performing second text vectorization processing on each of the answer texts through a pre-trained second text vectorization model to obtain multiple answer text vectors; The multiple answer text vectors are clustered to obtain multiple answer text clusters.

8. The method according to claim 7, characterized in that The clustering of the answer texts in the multiple groups of question and answer texts to obtain multiple answer text clusters is achieved by a second text vectorization model. Before training the first text vectorization model, the method further includes: Performing importance evaluation on the network parameters of each network layer of the second text vectorization model to obtain importance parameters of each network layer; According to a preset pruning ratio and the importance parameter, determining a network layer to be pruned, and removing the network layer to be pruned from the second text vectorization model to obtain a pruned second text vectorization model; The pruned second text vectorization model is used as the first text vectorization model to be trained.

9. The method according to claim 7, characterized in that: The clustering of the answer texts in the multiple groups of question and answer texts to obtain multiple answer text clusters is achieved through a second text vectorization model, the parameter amount of the first text vectorization model is smaller than the parameter amount of the second text vectorization model, and the first text vectorization model is a randomly initialized model.

10. The method according to any one of claims 1 to 5, characterized in that: After clustering the answer texts in the multiple groups of question-answer texts to obtain multiple answer text clusters, the method further includes: Determine the intra-cluster similarity of each of the answer text clusters, wherein the intra-cluster similarity represents the similarity between the answer texts in the answer text cluster; The answer text clusters whose intra-cluster similarity is less than or equal to an intra-cluster similarity threshold are filtered out from the plurality of question text clusters.

11. The method according to any one of claims 1 to 5, characterized in that: Before acquiring at least one question positive sample different from the first question text from the first question text cluster where the first question text is located, the method further includes: Obtain the number of texts in each of the problem text clusters; Based on the question text cluster whose text quantity is greater than or equal to the text quantity threshold, the step of obtaining at least one question positive sample different from the first question text from the first question text cluster where the first question text is located is performed.

12. The method according to any one of claims 1 to 5, characterized in that: In the case where the multiple groups of question and answer texts correspond to different fields, clustering the answer texts in the multiple groups of question and answer texts to obtain multiple answer text clusters includes: Dividing the multiple groups of question-answer texts into multiple sets, wherein each set corresponds to one of the fields; For each of the sets, clustering the answer texts in the set to obtain a plurality of answer text clusters in the field corresponding to the set; The step of training a first text vectorization model based on the plurality of first question texts, the question positive samples and the question negative samples respectively corresponding to the plurality of first question texts, and obtaining a trained first text vectorization model includes: For each of the sets, based on the multiple first question texts in the set and the question positive samples and the question negative samples respectively corresponding to the multiple first question texts, a first text vectorization model of the field corresponding to the set is trained to obtain a trained first text vectorization model of the field corresponding to the set.

13. The method according to claim 12, characterized in that In the case where the multiple groups of question-answer texts correspond to the first field, the method further includes: Based on multiple groups of question and answer texts in a second field, the first text vectorization model is fine-tuned to obtain a first text vectorization model in the second field, wherein the first field is different from the second field.

14. The method according to claim 13, characterized in that The step of fine-tuning the first text vectorization model based on the multiple groups of question-answer texts in the second field to obtain the first text vectorization model in the second field includes: Performing first text vectorization processing on the plurality of first question texts in the second field, and the problem positive samples and problem negative samples respectively corresponding to the plurality of first question texts in the second field by the first text vectorization model, to obtain first question text vectors of the first question texts in the second field, positive sample vectors of the problem positive samples in the second field, and negative sample vectors of the problem negative samples in the second field; Determine a second loss value based on the first question text vector of the second field, the positive sample vector of the second field, and the negative sample vector of the second field by using a preset loss function; Generate a first low-rank matrix initialized based on a Gaussian distribution, and generate a second low-rank matrix initialized based on all zeros; Constructing a fine-tuning weight parameter based on the first low-rank matrix and the second low-rank matrix, and adding the fine-tuning weight parameter to the network parameters of the first text vectorization model; The fine-tuning weight parameter is updated using the second loss value to obtain a first text vectorization model for the second domain.

15. A text processing method, characterized in that: The method comprises: Get multiple object question texts; Vectorizing the multiple object question texts through a first text vectorization model to obtain multiple object question text vectors, wherein the first text vectorization model is trained by the method described in any one of claims 1 to 14; Clustering the multiple object question text vectors to obtain at least one object question text cluster; The answer pre-associated with each of the object question text clusters is obtained as an answer in response to the object question text in each of the object question text clusters.

16. A training device for a text vectorization model, characterized in that: The device comprises: A data processing module, used for clustering answer texts in multiple groups of question and answer texts to obtain multiple answer text clusters, and constructing a question text cluster corresponding to each answer text cluster, wherein each group of question and answer texts includes a question text and the answer text corresponding to the question text; The data processing module is further used to obtain at least one question positive sample different from the first question text from the first question text cluster where the first question text is located, wherein the first question text is any one of the question texts; The data processing module is further used to obtain at least one negative question sample from a second question text cluster, wherein the second question text cluster is any question text cluster different from the first question text cluster; A training module is used to train a first text vectorization model based on a plurality of the first question texts, the question positive samples and the question negative samples respectively corresponding to the plurality of the first question texts, so as to obtain a trained first text vectorization model.

17. A text processing device, characterized in that: The device comprises: A data acquisition module is used to acquire multiple object question texts; A data processing module, configured to perform vectorization processing on the plurality of object question texts through a first text vectorization model to obtain a plurality of object question text vectors, wherein the first text vectorization model is trained by the method according to any one of claims 1 to 14; The data processing module is further used to perform clustering processing on the multiple object question text vectors to obtain at least one object question text cluster; The data processing module is further used to obtain an answer pre-associated with each of the object question text clusters as an answer in response to the object question text in each of the object question text clusters.

18. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, for implementing the text vectorization model training method described in any one of claims 1 to 14, or implementing the text processing method described in claim 15, when executing the computer executable instructions or computer programs stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, they implement the training method of the text vectorization model described in any one of claims 1 to 14, or implement the text processing method described in claim 15.

20. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, they implement the training method of the text vectorization model described in any one of claims 1 to 14, or implement the text processing method described in claim 15.

Images