Model training method and device, equipment, storage medium and product

Abstract

The embodiment of the invention discloses a model training method and device, equipment, a storage medium and a product. The method comprises the steps that first mask data and second mask data are obtained, the first mask data and the second mask data are obtained by conducting mask processing on elements in a first object through different mask strategies, first element prediction is conducted on the first mask data through an encoder, and second element prediction is conducted on the second mask data through an encoder; performing first element prediction on the first mask data to obtain a first prediction result and a feature vector of the first mask data, performing second element prediction on the feature vector of the first mask data and the second mask data by adopting a decoder to obtain a second prediction result, and performing joint training on the encoder and the decoder based on the first prediction result, the second prediction result and the first object to obtain a second prediction result. And obtaining the trained encoder. Therefore, through joint training of the encoder and the decoder, the encoder can be promoted to provide the more accurate feature vector of the first mask data to the decoder, and the object representation capability of the encoder is further improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, a model training device, a computer device, a computer-readable storage medium, and a model training product. Background Art

[0002] With advancements in scientific research, machine learning technology is developing rapidly. Models trained using machine learning techniques are widely used in our daily lives. For example, intelligent question-answering models can provide responses to questions posed; search engines can also output the content users are looking for based on the semantics of input data (prompt information). Research has found that the accuracy of model output results is positively correlated with the model's object representation capabilities, making improving these capabilities a hot research topic. Summary of the Invention

[0003] The embodiments of the present application provide a model training method, apparatus, device, computer-readable storage medium, and product that can improve the object representation capability of the model.

[0004] On the one hand, an embodiment of the present application provides a model training method, comprising:

[0005] Obtaining first mask data and second mask data, where the first mask data and the second mask data are obtained by masking elements in the first object using different masking strategies;

[0006] Performing a first element prediction on the first mask data using an encoder to obtain a first prediction result and a feature vector of the first mask data, wherein the first prediction result includes a prediction result of the masked element in the first mask data;

[0007] Using a decoder to perform second element prediction on the feature vector of the first mask data and the second mask data to obtain a second prediction result, where the second prediction result includes a prediction result of the masked element in the second mask data;

[0008] Based on the first prediction result, the second prediction result and the first object, the encoder and the decoder are jointly trained to obtain a trained encoder, and the trained encoder is used to predict the completion data corresponding to the prompt information.

[0009] In one aspect, an embodiment of the present application provides a model training device, comprising:

[0010] an acquiring unit, configured to acquire first mask data and second mask data, wherein the first mask data and the second mask data are obtained by masking elements in the first object using different masking strategies;

[0011] a processing unit, configured to perform a first element prediction on the first mask data using an encoder to obtain a first prediction result and a feature vector of the first mask data, wherein the first prediction result includes a prediction result of a masked element in the first mask data;

[0012] and performing a second element prediction on the feature vector of the first mask data and the second mask data using a decoder to obtain a second prediction result, the second prediction result including a prediction result of the masked element in the second mask data;

[0013] And it is used to jointly train the encoder and the decoder based on the first prediction result, the second prediction result and the first object to obtain a trained encoder, and the trained encoder is used to predict the completion data corresponding to the prompt information.

[0014] In one embodiment, the processing unit is configured to obtain the first mask data and the second mask data, specifically to:

[0015] Get a first object, where the first object includes M elements, where M is an integer greater than 1;

[0016] Performing random masking on the first object according to a first ratio to obtain first mask data;

[0017] Perform importance perception processing on M elements to obtain the average mutual information of the M elements. The average mutual information of the i-th element is used to indicate the importance of the i-th element in the first object, where i is a positive integer less than or equal to M.

[0018] Key element masking is performed on the first object according to the second ratio and the average mutual information of the M elements to obtain second mask data, where the second ratio is greater than the first ratio.

[0019] In one embodiment, the first object is text data, and the elements are characters or words in the text data; the processing unit is configured to perform importance perception processing on M elements to obtain average mutual information of the M elements, specifically for:

[0020] Calculate the mutual information of the i-th element based on the number of occurrences of the i-th element in the first object and the number of occurrences of the i-th element's neighboring elements in the first object;

[0021] Perform mean processing on the mutual information of the i-th element to obtain the average mutual information of the i-th element.

[0022] In one embodiment, the average mutual information of each element includes an importance score, and the importance score of the i-th element is proportional to the importance of the i-th element in the first object; the processing unit is configured to perform key element masking on the first object according to the second ratio and the average mutual information of the M elements to obtain second mask data, specifically for:

[0023] Calculate the ratio of the importance score of each element to the sum of the importance scores of M elements to obtain the mask probability of each element;

[0024] Calculate the number of masked elements of the first object according to the second ratio;

[0025] Based on the number of masked elements of the first object and the masking probabilities of the M elements, masking processing is performed on the first object to obtain second mask data.

[0026] In one embodiment, the first object is text data; the processing unit is configured to obtain the first object, specifically to:

[0027] Get the data to be processed;

[0028] Normalize the data to be processed to obtain at least one standardized text;

[0029] Identifiers are added to the start and end positions of each specification text to obtain the first object.

[0030] In one embodiment, the processing unit is configured to perform normalization processing on the data to be processed to obtain at least one normalized text, specifically for:

[0031] If there are non-text elements in the data to be processed, the data to be processed is cleaned to obtain the text data corresponding to the data to be processed;

[0032] If the encoding method of the text data is different from the preset encoding method, the encoding method of the text data is converted according to the preset encoding method to obtain text data encoded according to the preset encoding method;

[0033] Based on the text screening rules, the text data encoded according to the preset encoding method is screened to obtain at least one standard text;

[0034] The text screening rule includes at least one of the following: removing duplicate texts with a similarity higher than a similarity threshold, and removing erroneous texts with an error rate higher than an error threshold.

[0035] In one embodiment, the second mask data includes a start identifier; the processing unit is configured to use a decoder to perform a second element prediction on the feature vector of the first mask data and the second mask data to obtain a second prediction result, specifically for:

[0036] Replacing the start identifier of the second mask data with the feature vector of the first mask data to obtain updated second mask data;

[0037] A decoder is used to perform second element prediction on the updated second mask data to obtain a second prediction result.

[0038] In one embodiment, the processing unit is configured to jointly train the encoder and the decoder based on the first prediction result, the second prediction result, and the first object to obtain a trained encoder, specifically to:

[0039] constructing a first loss function based on a difference between a prediction result of a masked element in the first mask data and a corresponding element in the first object;

[0040] constructing a second loss function based on the difference between the prediction results of the masked elements in the second mask data and the corresponding elements in the first object;

[0041] Perform weighted summation on the first loss function and the second loss function to obtain a third loss function;

[0042] The encoder and decoder are jointly trained through the third loss function to obtain the trained encoder.

[0043] In one embodiment, the processing unit is further configured to:

[0044] Call the trained encoder to encode N candidate objects to obtain vector representations of N candidate objects, where N is an integer greater than 1;

[0045] Obtain prompt information and call the trained encoder to encode the prompt information to obtain a vector representation of the prompt information;

[0046] Based on the similarity between the vector representation of the prompt information and the vector representation of each candidate object, a second object is selected, wherein the similarity between the vector representation of the second object and the vector representation of the prompt information is greater than a similarity threshold;

[0047] The second object is determined as the completion data corresponding to the prompt information, and the second object is output.

[0048] In one embodiment, the processing unit is further configured to:

[0049] Call the trained encoder to encode N candidate objects to obtain vector representations of N candidate objects, each candidate object is associated with a reply message, and N is an integer greater than 1;

[0050] Obtain prompt information and call the trained encoder to encode the prompt information to obtain a vector representation of the prompt information;

[0051] Based on the similarity between the vector representation of the prompt information and the vector representation of each candidate object, a second object is selected, wherein the similarity between the vector representation of the second object and the vector representation of the prompt information is greater than a similarity threshold;

[0052] The reply information associated with the second object is determined as the completion data corresponding to the prompt information, and the reply information associated with the second object is output.

[0053] In one embodiment, the encoder includes P neural network structures, and the decoder includes Q neural network structures, where Q is a positive integer and P is an integer greater than Q; each neural network structure includes a multi-head self-attention mechanism layer and a feedforward neural network layer, and the multi-head self-attention mechanism layer and the feedforward neural network layer are respectively associated with a residual connection layer and a normalization layer; the feature vector of the first mask data is the output of the last neural network structure among the P neural network structures.

[0054] Accordingly, the present application provides a computer device, comprising:

[0055] a memory, wherein a computer program is stored in the memory;

[0056] A processor is used to load a computer program to implement the above-mentioned model training method.

[0057] Accordingly, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned model training method.

[0058] Accordingly, the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the above-mentioned model training method.

[0059] In an embodiment of the present application, first mask data and second mask data are obtained, and the first mask data and the second mask data are obtained by masking elements in a first object using different masking strategies. An encoder is used to predict the first element of the first mask data to obtain a first prediction result and a feature vector of the first mask data. A decoder is used to predict the second element of the feature vector of the first mask data and the second mask data to obtain a second prediction result. Based on the first prediction result, the second prediction result, and the first object, the encoder and decoder are jointly trained to obtain a trained encoder. It can be seen that by jointly training the encoder and decoder, the encoder can be prompted to provide the decoder with a more accurate feature vector of the first mask data, further improving the encoder's object representation capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 A data processing scenario diagram provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0063] Figure 3 A flowchart of another model training method provided in an embodiment of the present application;

[0064] Figure 4a A model architecture diagram provided for an embodiment of the present application;

[0065] Figure 4b A similarity analysis flow chart provided in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0069] This application involves technologies related to artificial intelligence. The following is a brief introduction to the related technologies involved:

[0070] Artificial Intelligence (AI): The so-called AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. The embodiment of the present application mainly involves performing semantic analysis on prompt information through a trained encoder to obtain the completion data corresponding to the prompt information.

[0071] AI technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0072] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning. The pre-training model is the latest development in deep learning and integrates the above technologies. The embodiment of the present application mainly involves jointly training the encoder and decoder based on the first mask data, the second mask data, and the first object to further enhance the encoder's object representation capabilities.

[0073] Based on the above-mentioned artificial intelligence-related technologies, an embodiment of the present application provides a model training solution that can improve the object representation capability of the model (encoder). Figure 1 A data processing scenario diagram provided in an embodiment of the present application, such as Figure 1As shown, the data processing scenario provided by the present application includes a terminal device 101 and a server 102, and the model training scheme provided by the present application can be executed by the server 102. Among them, the terminal device may include but is not limited to: smart phones (such as Android phones, IOS phones, etc.), tablet computers, portable personal computers, mobile Internet devices (Mobile Internet Devices, referred to as MID), intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc., and the embodiments of the present application do not limit this; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and basic cloud computing services such as big data and artificial intelligence platforms, and the embodiments of the present application do not limit this.

[0074] It should be noted that Figure 1 The number of terminal devices and servers is only used as an example and does not constitute an actual limitation of this application; for example, Figure 1 The system may also include a database for providing training data, or other terminal devices. In practical applications, the trained encoder can be directly installed in the terminal device 101. In this case, the server 102 may not be included in the data processing scenario. The terminal device 101 and the server 102 can be connected via a wired or wireless connection, which is not limited in this application.

[0075] The data processing involved in this application generally follows the following process: the server 102 encodes the candidate objects in advance (through a trained encoder) to obtain a vector representation of each candidate object. The candidate object can specifically be text data (such as a sentence, a title, etc.), or image data (such as a picture), or multimedia data (such as video, audio, files, etc.). This application does not limit this. When the server 102 obtains the prompt information (such as prompt words, keywords or keywords, etc.) sent by the terminal device 101, the server 102 calls the trained encoder to encode the prompt information to obtain a vector representation of the prompt information. The server 102 then calculates the vector representation of the prompt information and the similarity between it and the vector representation of each candidate object (such as cosine similarity, Euler distance, etc.), and can determine the candidate object with the highest similarity as the completion data corresponding to the prompt information. For example, in a search scenario, the server 102 can output search results (such as files, short videos, pictures, etc.) based on the search terms entered by the user; for another example, in a question-and-answer scenario, the server 102 can determine the candidate object (question) that is most similar to the question entered by the user based on the similarity, and generate answer information for the question entered by the user based on the answer information of the candidate object.

[0076] In the above data processing process, the training process of the trained encoder involved is as follows:

[0077] (1) The server 102 obtains first mask data and second mask data, which are obtained by masking elements in a first object using different masking strategies; the first object may be text or an image. In one embodiment, the first object is a sentence, and the elements may refer to words or characters in the sentence; in another embodiment, the first object is a picture, and the elements may refer to partial images after the picture is divided. The masking strategies may include, but are not limited to: a random masking strategy, a preset rule masking strategy (such as masking according to a preset interval), an importance masking strategy, a frequency masking strategy, and a content masking strategy (such as masking based on the amount of content contained in each partial image).

[0078] In one embodiment, the server 102 obtains a first object, where the first object includes M elements, where M is an integer greater than 1. On the one hand, the server 102 performs a random masking process on the first object according to a first ratio (e.g., randomly replacing j elements of the M elements with preset identifiers, where j is calculated based on M and the first ratio) to obtain first masked data. On the other hand, the server 102 performs an importance masking process on the first object according to a second ratio to obtain second masked data, where the second ratio is greater than the first ratio.

[0079] (2) The server 102 uses an encoder to predict the first element of the first mask data to obtain a first prediction result and a feature vector of the first mask data, and uses a decoder to predict the second element of the feature vector of the first mask data and the second mask data to obtain a second prediction result. In one embodiment, both the encoder and the decoder include neural network structures, and the model structure of the encoder is more complex than the model structure of the decoder (e.g., the encoder includes more neural network structures than the decoder). It can be understood that the ability of the encoder is better than that of the decoder; for example, compared with the decoder, the encoder can use more neural network structures to mine deeper features of the input data and obtain a more accurate vector representation of the input data.

[0080] (3) The server 102 jointly trains the encoder and decoder based on the first prediction result, the second prediction result and the first object to obtain a trained encoder, and the trained encoder is used to predict the complementary data corresponding to the prompt information. In one embodiment, on the one hand, the server 102 constructs a first loss function based on the difference between the prediction result of the masked element in the first mask data and the corresponding element in the first object; on the other hand, the server 102 constructs a second loss function based on the difference between the prediction result of the masked element in the second mask data and the corresponding element in the first object. After obtaining the first loss function and the second loss function, the server 102 performs a weighted summation process on the first loss function and the second loss function to obtain a third loss function. The encoder and decoder are jointly trained using the third loss function (such as by adjusting the model parameters in the encoder and decoder so that the value of the third loss function is less than the loss threshold) to obtain a trained encoder.

[0081] In an embodiment of the present application, first mask data and second mask data are obtained, and the first mask data and the second mask data are obtained by masking the elements in the first object using different masking strategies. An encoder is used to predict the first element of the first mask data to obtain a first prediction result and a feature vector of the first mask data. A decoder is used to predict the second element of the feature vector of the first mask data and the second mask data to obtain a second prediction result. Based on the first prediction result, the second prediction result and the first object, the encoder and the decoder are jointly trained to obtain a trained encoder. It can be seen that by jointly training the encoder and the decoder, the encoder can be prompted to provide the decoder with a more accurate feature vector of the first mask data, further improving the object representation capability of the encoder. In addition, by using the trained encoder to predict the completion information corresponding to the prompt information, the accuracy of the prediction result can be further improved.

[0082] Based on the above-mentioned model training scheme, the embodiment of the present application proposes a more detailed model training method. The model training method proposed in the embodiment of the present application will be introduced in detail below with reference to the accompanying drawings.

[0083] See also Figure 2 , Figure 2 A flowchart of a model training method provided in an embodiment of the present application, the model training method can be executed by a computer device; for example, Figure 1 The server 102 shown in FIG.

[0084] like Figure 2 As shown, the model training method may include the following steps S201-S204:

[0085] S201: Acquire first mask data and second mask data.

[0086] The first mask data and the second mask data are obtained by masking elements in the first object using different masking strategies. The first object can be text or an image. When the first object is text, the elements can be characters or words in the text. When the first object is an image, the elements can be local images in the image. For example, the image can be divided into a grid to obtain multiple sub-images contained in the image, and the image can be masked using corresponding masking strategies (such as replacing the sub-images with blank images of a preset color) to obtain mask data.

[0087] The masking strategy may include but is not limited to: random masking strategy, preset rule masking strategy (such as masking according to preset intervals), importance masking strategy, frequency masking strategy, content masking strategy (such as masking based on the amount of content contained in each local image).

[0088] In one embodiment, a computer device obtains a first object, where the first object includes M elements, where M is an integer greater than 1. The computer device may mask the first object using the same ratio and different masking methods (e.g., random masking and importance masking) to obtain first masked data and second masked data. The computer device may also mask the first object using the same masking method at different ratios to obtain first masked data and second masked data. The computer device may also mask the first object using different ratios and different masking methods to obtain first masked data and second masked data.

[0089] In one embodiment, on the one hand, a computer device performs random masking on a first object according to a first ratio (e.g., randomly replacing j elements out of M elements with preset identifiers, where j is calculated based on M and the first ratio) to obtain first masked data; on the other hand, the computer device performs importance masking on the first object according to a second ratio (where the second ratio is greater than the first ratio) to obtain second masked data. Specifically, the computer device calculates the Pointwise Mutual Information (PMI) of the i-th element based on the number of occurrences of the i-th element in the first object and the number of occurrences of the i-th element's neighboring elements in the first object; the i-th element's neighboring elements include at least one of the i-1-th element and the i+1-th element, where i is a positive integer less than or equal to M. After obtaining the mutual information of the i-th element, the computer device performs mean processing on the mutual information of the i-th element to obtain the average mutual information (Average Mutual Information, AMI) of the i-th element. The average mutual information of the i-th element is used to indicate the importance of the i-th element in the first object. The average mutual information of each element includes an importance score, and the importance score of the i-th element is proportional to the importance of the i-th element in the first object. After obtaining the average mutual information of M elements, the computer device calculates the ratio of the importance score of each element to the sum of the importance scores of the M elements to obtain the mask probability of each element; calculates the number of masked elements of the first object according to the second ratio; and performs mask processing on the first object based on the number of masked elements of the first object and the mask probability of the M elements to obtain second mask data.

[0090] In another embodiment, after obtaining the importance score of each element in the above manner, the computer device selects k mask elements from the M elements based on the second ratio and the importance scores of the M elements. These k mask elements are the k elements with the highest importance scores among the M elements, where k is a positive integer less than M. The computer device then performs masking processing on the k mask elements in the first object to obtain second mask data.

[0091] S202: Use an encoder to perform first element prediction on the first mask data to obtain a first prediction result and a feature vector of the first mask data.

[0092] The first prediction result includes the prediction result of the masked element in the first mask data. The feature vector of the first mask data is obtained based on the first mask data and can be understood as the predicted feature vector of the first object. It can be understood that the closer the feature vector of the first mask data is to the feature vector of the first object, the more conducive the execution of step S203 is.

[0093] In one embodiment, the encoder includes P neural network structures (transformer model structures), where P is an integer greater than 1. Each neural network structure includes a multi-head self-attention mechanism layer and a feed-forward neural network layer, and each of the multi-head self-attention mechanism layer and the feed-forward neural network layer is associated with a residual connection layer and a normalization layer. The feature vector of the first mask data is the output of the last neural network structure arranged (in the data flow direction) among the P neural network structures.

[0094] S203 : Use a decoder to perform second element prediction on the feature vector of the first mask data and the second mask data to obtain a second prediction result.

[0095] The second prediction result includes the prediction result of the masked element in the second mask data. In one embodiment, the decoder includes Q neural network structures, where Q is a positive integer less than P. It should be noted that the network structures of the above-mentioned encoder and decoder are only used for example and do not constitute an actual limitation of this application. For example, the residual connection layer or the normalization layer is shared. For another example, the encoder and decoder may also include other neural network layers, which can be adjusted according to the object, such as using different neural network layer structures for text and images.

[0096] In one embodiment, the second mask data includes a start identifier. For example, if the second mask data is a sentence, since the decoder is not required to predict the prompt information after training, the second mask data does not need to be used as a sentence vector. Therefore, the start identifier can be omitted from the second mask data. The computer device replaces the start identifier of the second mask data with the feature vector of the first mask data to obtain updated second mask data. The computer device then uses the decoder to perform a second element prediction on the updated second mask data to obtain a second prediction result.

[0097] Optionally, the computer device may keep the second mask data unchanged, add the feature vector of the first mask data before or after the second mask data, and then use a decoder to perform a second element prediction on the second mask data after adding the feature vector of the first mask data to obtain a second prediction result.

[0098] S204 : Jointly train the encoder and decoder based on the first prediction result, the second prediction result, and the first object to obtain a trained encoder.

[0099] A trained encoder is obtained, and the trained encoder is used to predict the completion data corresponding to the prompt information. The completion data may include but is not limited to: the complete content corresponding to the prompt information, the reply content corresponding to the prompt information, and the file or multimedia data corresponding to the prompt information.

[0100] In one embodiment, the computer device constructs a first loss function based on the difference between the prediction result of the masked element in the first mask data and the corresponding element in the first object, and constructs a second loss function based on the difference between the prediction result of the masked element in the second mask data and the corresponding element in the first object. After obtaining the first loss function and the second loss function, the computer device performs weighted summation processing on the first loss function and the second loss function to obtain a third loss function, and jointly trains the encoder and decoder through the third loss function (such as by adjusting the model parameters in the encoder and decoder so that the value of the third loss function is less than the loss threshold) to obtain a trained encoder. The loss function can specifically be a cross entropy loss function, a multi-label classification loss function, a mean square loss function, etc., and this application does not limit this.

[0101] In an embodiment of the present application, first mask data and second mask data are obtained, and the first mask data and the second mask data are obtained by masking elements in a first object using different masking strategies. An encoder is used to predict the first element of the first mask data to obtain a first prediction result and a feature vector of the first mask data. A decoder is used to predict the second element of the feature vector of the first mask data and the second mask data to obtain a second prediction result. Based on the first prediction result, the second prediction result, and the first object, the encoder and decoder are jointly trained to obtain a trained encoder. It can be seen that by jointly training the encoder and decoder, the encoder can be prompted to provide the decoder with a more accurate feature vector of the first mask data, further improving the encoder's object representation capability.

[0102] See also Figure 3 , Figure 3 A flowchart of another model training method provided in an embodiment of the present application, which can be executed by a computer device; for example, Figure 1 The server 102 shown in FIG.

[0103] like Figure 3 As shown, the model training method may include the following steps S301-S309:

[0104] S301: Acquire a first object.

[0105] The first object includes M elements, where M is an integer greater than 1. The first object can be text or an image. When the first object is text, the elements can be characters or words in the text. When the first object is an image, the elements can be local images in the image; for example, the image can be divided into a grid to obtain multiple sub-images contained in the image, and the image can be masked using a corresponding masking strategy (such as replacing the sub-image with a blank image of a preset color) to obtain mask data.

[0106] In one embodiment, the object is text. The computer device obtains the text to be processed and normalizes the text to be processed to obtain at least one normalized text. The specific process includes: if there are non-text elements (such as emoticons, special symbols, etc.) in the data to be processed, the computer device cleans the data to be processed (such as removing emoticons in the text) to obtain text data corresponding to the data to be processed. If the encoding method of the text data is different from the preset encoding method (such as UTF-8), the encoding method of the text data is converted according to the preset encoding method to obtain text data encoded according to the preset encoding method. In addition, the computer device can also filter the text data encoded according to the preset encoding method based on the text filtering rules to obtain at least one normalized text; wherein the text filtering rules include at least one of the following: removing duplicate texts with a similarity higher than a similarity threshold, and removing erroneous texts with an error rate higher than an error threshold. For example, the computer device calculates the hash value of each text data and adds the hash value to a hash table. If there is text data with the same hash value in the hash table, the text data is determined to be duplicate text. For another example, the computer device can perform error detection on each text data (such as spelling errors, grammatical errors, logical errors, etc.). If the error rate of a text data (such as the ratio of the number of incorrect words to the total number of words in the text data) is higher than the error threshold, the text data is determined to be erroneous text.

[0107] In one embodiment, after obtaining at least one canonical text, the computer device may add identifiers to the start and end positions of each canonical text, and perform word segmentation and indexing on the canonical text to obtain a first object. For example, if each canonical text is a sentence, the computer device may add the identifier cls at the beginning of the sentence and the identifier sep at the end of the sentence, and perform word segmentation and indexing on the canonical text to obtain a first object. In another embodiment, after obtaining at least one canonical text, the computer device may directly perform word segmentation and indexing on the canonical text to obtain the first object (i.e., without adding identifiers).

[0108] In another embodiment, the object is an image. The computer device acquires the image to be processed, and performs normalization processing on the image to be processed to obtain at least one normalized image. The specific process may include at least one of the following: performing scaling processing on the image to be processed, performing cropping processing on the image to be processed (such as removing meaningless solid-color regions in the image), and performing screening on the image to be processed (such as removing duplicate pictures, pictures with clarity lower than the clarity threshold, etc.). Then the computer device may use the normalized image as the first object.

[0109] S302. Perform random masking processing on the first object according to the first ratio to obtain first mask data.

[0110] For example, assume that the first ratio is 20%, and the first object includes 10 elements. Then the computer device randomly selects two elements from the 10 elements and replaces the selected two elements with a preset identifier (such as mask) to obtain the first mask data.

[0111] It should be noted that the random masking strategy ignores the importance of different words and treats all words in the text equally. In the text corpus, different words contain different amounts of information. For example, words such as "of", "and", and "le" appear very frequently but contain very little information; on the contrary, entity words such as "a certain place" and "XX building" appear less frequently but contain a lot of information. Using the random masking strategy, it is more likely to mask out high-frequency low-information words, and then these high-frequency words with very little information contribute very little to generating dense sentence vector representations. If the second mask data is obtained using the same random masking strategy and the decoder is used to perform second element prediction on the feature vector of the first mask data and the second mask data, it reduces the difficulty of the decoder to recover and predict, which is not conducive to the encoder outputting a high-quality feature vector of the first mask data (there may be a situation where even if the feature vector of the first mask data output by the encoder is inaccurate, the decoder can accurately obtain the second prediction result).

[0112] S303. Perform importance perception processing on the M elements to obtain the average mutual information of the M elements.

[0113] In one embodiment, the first object is text data. The computer device calculates the mutual information of the i-th element based on the number of occurrences of the i-th element in the first object and the number of occurrences of the adjacent elements of the i-th element in the first object, where i is a positive integer less than or equal to M. The adjacent elements of the i-th element refer to the elements whose distance from the i-th element in the first object is less than or equal to the distance range. In one embodiment, the distance range is 1, and the adjacent elements of the i-th element include at least one of the i-1th element and the i+1th element. When i=1, the adjacent elements of the i-th element only include the 2nd element; when i=M, the adjacent elements of the i-th element only include the M-1th element. The calculation formula of the mutual information can be expressed as:

[0114]

[0115] Among them, PMI(x1,x2) represents the mutual information between elements x1 and x2, p(x1,x2) is the probability of the binary group (x1,x2), count(x1,x2) is the number of occurrences of the bigram (x1,x2) in the corpus (the first object), and N is the number of all bigrams in the corpus (the first object). Similarly, p(x1) is the probability of the unigram (x1), and p(x2) is the probability of the unigram (x2).

[0116] After obtaining the mutual information of the i-th element, the computer device performs mean processing on the mutual information of the i-th element to obtain the average mutual information of the i-th element. The average mutual information of the i-th element is used to indicate the importance of the i-th element in the first object. The calculation formula of the average mutual information of the i-th element can be expressed as:

[0117] AMI(x i )=aPMI(x i-1 ,x i )+bPMI(x i ,x i+1 )

[0118] Among them, AMI(x i ) represents the average mutual information of the i-th element, a and b are weights, a+b=1. i-1 ,x i ) represents the mutual information between the i-1th element and the i-th element, PMI(x i ,x i+1 ) represents the mutual information between the i-th element and the i+1-th element. According to the above embodiment, the computer device can calculate the average mutual information of the M elements.

[0119] In another embodiment, the first object is an image, and the average mutual information of each element includes an importance score. The importance score of each sub-image (element) can be determined by the position of the sub-image in the image and the amount of information carried by the sub-image; for example, the closer the sub-image is to the center of the image, the higher the importance score; for another example, the more information a sub-image carries, the higher the importance score.

[0120] S304 : Perform key element masking processing on the first object according to the second ratio and the average mutual information of the M elements to obtain second mask data.

[0121] In one embodiment, the average mutual information of each element includes an importance score, and the importance score of the i-th element is proportional to the importance of the i-th element in the first object (elements with importance scores higher than a score threshold can be regarded as key elements). The computer device calculates the ratio of the importance score of each element to the sum of the importance scores of the M elements to obtain the mask probability of each element. Specifically, it can be expressed as:

[0122]

[0123] Among them, p i represents the mask probability of the i-th element (indicates the probability that the i-th element is masked in one mask), t i is the importance score of the i-th element, ∑ j t j It represents the sum of the importance scores of M elements. According to the mask probability formula of each element, the higher the importance score of an element, the greater the mask probability of the element.

[0124] After calculating the mask probability of each element, the computer device calculates the number of masked elements of the first object according to the second ratio, and masks the first object based on the number of masked elements of the first object and the mask probability of the M elements to obtain second mask data.

[0125] Optionally, the computer device selects k mask elements from the M elements based on the second ratio and the average mutual information of the M elements, where the average mutual information of any of the k mask elements is greater than or equal to the average mutual information of non-masked elements, and the non-masked elements are elements in the M elements other than the k mask elements, where k is a positive integer less than M. After obtaining the k mask elements, the computer device performs masking processing on the k mask elements in the first object to obtain second mask data.

[0126] Exemplarily, the first object, the first mask data, and the second mask data may be represented as:

[0127] X = ['cls', 'a certain place', 'XX building', ', ', 'has a 'long', ',' culture', 'and ', 'history', '. ', 'sep']

[0128]

[0129]

[0130] Here, X is the first object (a sentence). The first mask data is first mask data. For example, the first mask data may be obtained by performing random masking processing on the first object according to a first ratio. is the second mask data. For example, the second mask data may be obtained by performing importance masking on the first object according to the second ratio (for details, see steps S303 and S304). cls is the start identifier (in this example, used to indicate the beginning of a sentence), sep is the end identifier (in this example, used to indicate the end of a sentence), and mask is a preset identifier used to replace the content in the sentence.

[0131] S305 : Use an encoder to perform first element prediction on the first mask data to obtain a first prediction result and a feature vector of the first mask data.

[0132] In one embodiment, the encoder includes multiple transformer layers. Taking text objects as an example, a trained pre-trained language model (such as BERT) can be used to initialize the encoder, so that the rich language knowledge learned by the pre-trained language model can be fully utilized. Each transformer layer contains two sublayers: a multi-head self-attention mechanism and a feed-forward neural network. In addition, each sublayer is followed by a residual connection and layer normalization. The multi-head self-attention mechanism calculates the degree of association between each word in the input sequence and other words, thereby capturing long-distance dependencies in the sentence. The multi-head mechanism allows the transformer layer to pay attention to information at different positions at the same time. The feed-forward neural network is used to extract local features of the input sequence and usually includes two fully connected layers and an activation function.

[0133] The encoder model uses a bidirectional attention mechanism. Taking text data as an example, the transformer layer can simultaneously consider the contextual information of a word (for images, the transformer layer can also consider information about the surrounding image), including the word before it and the word after it. The bidirectional attention mechanism enables the encoder model to better capture the semantic information in the context when understanding text. In the encoder model, the representation of each word is calculated based on the entire input text sequence, which enables the encoder model to better understand the meaning of each word in context.

[0134] In one embodiment, the computer device uses the first masked data (i.e., the perturbed text) as the input of the encoder model. The deep encoder is trained using the Masked Language Model (MLM) as the target. In the Masked Language Model target, the encoder model predicts the masked words (i.e., the first prediction result) based on the context vector representation. In the process of training the encoder, the computer device can use a cross-entropy loss function to measure the difference between the encoder predicted elements and the actual elements, and update the encoder parameters by minimizing the loss function, which can be specifically expressed as:

[0135]

[0136] Among them, L enc is the first loss function. i represents the i-th element, M enc Indicates the masked elements (replaced with preset identifiers) in the first mask data. is the first mask data. Indicates that when the input is The probability that the i-th element is predicted correctly when .

[0137] In addition, the computer device uses the vector output by the last transformer layer of the encoder (data flow order) as the feature vector of the first mask data; for example, when the first mask data is a masked sentence, the feature vector of the first mask data is the dense sentence vector representation.

[0138] S306 : Use a decoder to perform second element prediction on the feature vector of the first mask data and the second mask data to obtain a second prediction result.

[0139] In one embodiment, the decoder includes fewer transformer layers than the encoder (e.g., 1-3 transformer layers). In one embodiment, the computer device randomly initializes the model parameters of the decoder. The feature vector of the first mask data and the second mask data are used as the input of the decoder. The training goal of the decoder is to predict the masked elements based on the feature vector and the second mask data of the first mask data output by the deep encoder, and to reconstruct and restore the disturbed first object. During the training process of the decoder, the computer device also uses a cross entropy loss function to measure the difference between the decoder predicted elements and the real elements, and updates the decoder parameters by minimizing the loss function, which can be specifically expressed as:

[0140]

[0141] Among them, L dec is the second loss function. i represents the i-th element, M dec Indicates the masked element (replaced with a preset identifier) ​​in the second mask data, h x is the feature vector of the first mask data. is the second mask data. Indicates that when input h x and The probability that the i-th element is predicted correctly when .

[0142] S307 : Based on the first prediction result, the second prediction result, and the first object, jointly train the encoder and the decoder to obtain a trained encoder.

[0143] In one embodiment, the first prediction result and the first object are used to construct a first loss function. For a specific embodiment, refer to L in step S305. enc The construction method is not described here; similarly, the second prediction result and the first object are used to construct the second loss function. The specific implementation method can refer to L in step S306. dec The construction method of is not described here. After obtaining the first loss function and the second loss function, the computer device can perform weighted summation processing on the first loss function and the second loss function to obtain a third loss function, which can be specifically expressed as:

[0144] L=aL enc +bL dec

[0145] Among them, L is the third loss function, L enc is the first loss function, L dec is the second loss function, a and b are weights, a+b=1; for example, a=b=0.5.

[0146] Figure 4a This is a model architecture diagram provided in the embodiment of this application. Figure 4a As shown, the first mask data is input into the encoder to obtain the first prediction result and the feature vector of the first mask data. On the one hand, a first loss function is constructed based on the difference between the first prediction result and the first object; on the other hand, the starting identifier in the second mask data can be replaced with the feature vector of the first mask data to obtain updated second mask data. The updated second mask data is input into the decoder to obtain the second prediction result, and a second loss function is constructed based on the difference between the second prediction result and the first object. After obtaining the first loss function and the second loss function, the first loss function and the second loss function are weighted and summed to obtain a third loss function, and the encoder and decoder are jointly trained using the third loss function to obtain the trained encoder and decoder.

[0147] It should be noted that compared to the encoder, the decoder contains fewer transformer layers, a simpler model structure, and weaker model capabilities (such as result prediction accuracy); in addition, the encoder adopts a random masking strategy with a small mask ratio, while the decoder adopts an importance-aware masking strategy with a large mask ratio, which increases the prediction difficulty of the decoder. In order to obtain more accurate prediction results, the decoder needs to pay attention to the feature vector of the first mask data output by the encoder. This forces the encoder to capture enough object features (such as the semantic information of the input text) and output high-quality dense vectors. In other words, during the training process, this application improves the object representation ability of the encoder by increasing the prediction difficulty of the decoder.

[0148] S308: Obtain prompt information.

[0149] The prompt information can vary with the application scenario and can be a title, a keyword or a word, a question, etc., which is not limited in this application. For example, in a file retrieval scenario, the prompt information can be a title; in another example, in a question-and-answer scenario, the prompt information can be a question.

[0150] S309: Predict the completion data corresponding to the prompt information through the trained encoder.

[0151] The completion data corresponds to the prompt information and can be text, multimedia data (such as a short video), reply information, etc. This application does not impose any restrictions on this. For example, in a text search scenario, the prompt information can be a keyword, and the completion data can be the corresponding sentence, paragraph, or article; for another example, in a question-and-answer scenario, the prompt information can be a question, and the completion data can be a reply information.

[0152] In one embodiment, on the one hand, the computer device calls the trained encoder to encode N candidate objects to obtain vector representations of the N candidate objects, where N is an integer greater than 1; on the other hand, the computer device calls the trained encoder to encode the prompt information to obtain the vector representation of the prompt information. After obtaining the vector representation of the prompt information and the vector representations of the N candidate objects, the computer device screens out the second object based on the similarity between the vector representation of the prompt information and the vector representation of each candidate object. The similarity between the vector representation of the second object and the vector representation of the prompt information is greater than a similarity threshold. The similarity threshold can be static or dynamic. The dynamic similarity threshold can be used to screen the candidate object whose vector representation has the highest similarity with the vector representation of the prompt information among the N candidate objects. The number of second objects can be one or more. When there are multiple second objects, the computer device can sort the second objects in descending order of similarity.

[0153] Figure 4b A similarity analysis flow chart provided in the embodiment of this application. Figure 4b As shown in Figure 2, the similarity analysis process can be divided into the indexing stage and the retrieval stage. In the indexing stage, the computer device encodes N candidate objects into fixed-dimensional object vector representations {p1, p2, ... p N During the retrieval phase, for an input prompt (such as a query), the computer device calls the trained encoder to encode the prompt into a target vector representation q. The cosine similarity score is then calculated for each of the target vector representation and the object vector representations of the N candidate objects. The score range can be limited to between 0 and 1. The specific calculation formula can be expressed as:

[0154]

[0155] Among them, score is the similarity score, q is the target vector representation, p i is the object vector representation of the i-th candidate object, i is a positive integer less than or equal to N. |q| is the modulus of the target vector representation q; |p i | is the object vector representation p of the i-th candidate object i The modulus of the cosine similarity score. A higher cosine similarity score indicates a greater similarity between the prompt information and the candidate object; conversely, a lower cosine similarity score indicates a lower similarity between the prompt information and the candidate object. The computer device may select the candidate object with the highest similarity score as the second object.

[0156] In one embodiment, after filtering out the second object, the computer device may directly determine the second object as the completion data corresponding to the prompt information, and output the second object.

[0157] In another embodiment, each candidate object is associated with a file (such as a text file, a multimedia file, etc.). After screening out the second object, the computer device can determine the file associated with the second object as the completion data corresponding to the prompt information and output the file associated with the second object.

[0158] In another embodiment, each candidate object is associated with a reply information. After screening out the second object, the computer device can determine the reply information associated with the second object as the completion data corresponding to the prompt information, and output the reply information associated with the second object; or, the computer device optimizes the reply information associated with the second object based on the prompt information, and determines the optimized result as the completion data corresponding to the prompt information, and outputs the optimized result.

[0159] In the embodiment of the present application, compared with the encoder, the decoder contains fewer transformer layers, the model structure is simpler, and the model capability is weaker (such as the result prediction accuracy); in addition, the encoder adopts a random masking strategy with a small mask ratio, while the decoder adopts an importance-aware masking strategy with a large mask ratio, which increases the prediction difficulty of the decoder. In order to obtain more accurate prediction results, the decoder needs to pay attention to the feature vector of the first mask data output by the encoder. This forces the encoder to capture enough object features (such as the semantic information of the input text) and output a high-quality dense vector. In other words, during the training process, the present application improves the object representation capability of the encoder by increasing the prediction difficulty of the decoder. In addition, the quality of the training data (first object) can be improved by cleaning the data to be processed; and the accuracy of the prediction results can be further improved by predicting the completion information corresponding to the prompt information through the trained encoder.

[0160] The above describes in detail the method of the embodiment of the present application. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0161] See Figure 5 , Figure 5 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Figure 5 The model training apparatus shown can be mounted in a computer device, which can specifically be a terminal device or a server. Figure 5 The model training apparatus shown can be used to perform the above Figure 2 and Figure 3 Some or all of the functions described in the method embodiments. Figure 5 , the model training device includes:

[0162] An acquiring unit 501 is configured to acquire first mask data and second mask data, where the first mask data and the second mask data are obtained by masking elements in a first object using different masking strategies.

[0163] A processing unit 502 is configured to perform a first element prediction on the first mask data using an encoder to obtain a first prediction result and a feature vector of the first mask data, where the first prediction result includes a prediction result of the masked element in the first mask data;

[0164] and performing a second element prediction on the feature vector of the first mask data and the second mask data using a decoder to obtain a second prediction result, the second prediction result including a prediction result of the masked element in the second mask data;

[0165] And it is used to jointly train the encoder and the decoder based on the first prediction result, the second prediction result and the first object to obtain a trained encoder, and the trained encoder is used to predict the completion data corresponding to the prompt information.

[0166] In one embodiment, the processing unit 502 is configured to obtain the first mask data and the second mask data, specifically to:

[0167] Get a first object, where the first object includes M elements, where M is an integer greater than 1;

[0168] Performing random masking on the first object according to a first ratio to obtain first mask data;

[0169] Perform importance perception processing on M elements to obtain the average mutual information of the M elements. The average mutual information of the i-th element is used to indicate the importance of the i-th element in the first object, where i is a positive integer less than or equal to M.

[0170] Key element masking is performed on the first object according to the second ratio and the average mutual information of the M elements to obtain second mask data, where the second ratio is greater than the first ratio.

[0171] In one embodiment, the first object is text data, and the elements are characters or words in the text data; the processing unit 502 is configured to perform importance perception processing on the M elements to obtain the average mutual information of the M elements, specifically for:

[0172] Calculate the mutual information of the i-th element based on the number of occurrences of the i-th element in the first object and the number of occurrences of the i-th element's neighboring elements in the first object;

[0173] Perform mean processing on the mutual information of the i-th element to obtain the average mutual information of the i-th element.

[0174] In one embodiment, the average mutual information of each element includes an importance score, and the importance score of the i-th element is proportional to the importance of the i-th element in the first object; the processing unit 502 is configured to perform key element masking on the first object according to the second ratio and the average mutual information of the M elements to obtain second mask data, specifically for:

[0175] Calculate the ratio of the importance score of each element to the sum of the importance scores of M elements to obtain the mask probability of each element;

[0176] Calculate the number of masked elements of the first object according to the second ratio;

[0177] Based on the number of masked elements of the first object and the masking probabilities of the M elements, masking processing is performed on the first object to obtain second mask data.

[0178] In one embodiment, the first object is text data; the processing unit 502 is configured to obtain the first object, specifically to:

[0179] Get the data to be processed;

[0180] Normalize the data to be processed to obtain at least one standardized text;

[0181] Identifiers are added to the start and end positions of each specification text to obtain the first object.

[0182] In one embodiment, the processing unit 502 is configured to perform normalization processing on the data to be processed to obtain at least one normalized text, specifically for:

[0183] If there are non-text elements in the data to be processed, the data to be processed is cleaned to obtain the text data corresponding to the data to be processed;

[0184] If the encoding method of the text data is different from the preset encoding method, the encoding method of the text data is converted according to the preset encoding method to obtain text data encoded according to the preset encoding method;

[0185] Based on the text screening rules, the text data encoded according to the preset encoding method is screened to obtain at least one standard text;

[0186] The text screening rule includes at least one of the following: removing duplicate texts with a similarity higher than a similarity threshold, and removing erroneous texts with an error rate higher than an error threshold.

[0187] In one embodiment, the second mask data includes a start identifier; the processing unit 502 is configured to use a decoder to perform a second element prediction on the feature vector of the first mask data and the second mask data to obtain a second prediction result, specifically for:

[0188] Replacing the start identifier of the second mask data with the feature vector of the first mask data to obtain updated second mask data;

[0189] A decoder is used to perform second element prediction on the updated second mask data to obtain a second prediction result.

[0190] In one embodiment, the processing unit 502 is configured to jointly train the encoder and the decoder based on the first prediction result, the second prediction result, and the first object to obtain a trained encoder, specifically to:

[0191] constructing a first loss function based on a difference between a prediction result of a masked element in the first mask data and a corresponding element in the first object;

[0192] constructing a second loss function based on the difference between the prediction results of the masked elements in the second mask data and the corresponding elements in the first object;

[0193] Perform weighted summation on the first loss function and the second loss function to obtain a third loss function;

[0194] The encoder and decoder are jointly trained through the third loss function to obtain the trained encoder.

[0195] In one embodiment, the processing unit 502 is further configured to:

[0196] Call the trained encoder to encode N candidate objects to obtain vector representations of N candidate objects, where N is an integer greater than 1;

[0197] Obtain prompt information and call the trained encoder to encode the prompt information to obtain a vector representation of the prompt information;

[0198] Based on the similarity between the vector representation of the prompt information and the vector representation of each candidate object, a second object is selected, wherein the similarity between the vector representation of the second object and the vector representation of the prompt information is greater than a similarity threshold;

[0199] The second object is determined as the completion data corresponding to the prompt information, and the second object is output.

[0200] In one embodiment, the processing unit 502 is further configured to:

[0201] Call the trained encoder to encode N candidate objects to obtain vector representations of N candidate objects, each candidate object is associated with a reply message, and N is an integer greater than 1;

[0202] Obtain prompt information and call the trained encoder to encode the prompt information to obtain a vector representation of the prompt information;

[0203] Based on the similarity between the vector representation of the prompt information and the vector representation of each candidate object, a second object is selected, wherein the similarity between the vector representation of the second object and the vector representation of the prompt information is greater than a similarity threshold;

[0204] The reply information associated with the second object is determined as the completion data corresponding to the prompt information, and the reply information associated with the second object is output.

[0205] In one embodiment, the encoder includes P neural network structures, and the decoder includes Q neural network structures, where Q is a positive integer and P is an integer greater than Q; each neural network structure includes a multi-head self-attention mechanism layer and a feedforward neural network layer, and the multi-head self-attention mechanism layer and the feedforward neural network layer are respectively associated with a residual connection layer and a normalization layer; the feature vector of the first mask data is the output of the last neural network structure among the P neural network structures.

[0206] According to one embodiment of the present application, Figure 2 and Figure 3 Some of the steps involved in the model training method shown can be represented by Figure 5 The model training device shown is executed by each unit. For example, Figure 2 The step S201 shown in FIG. Figure 5 The acquisition unit 501 shown in FIG. 5 is executed, and steps S202 to S204 can be performed by Figure 5 The processing unit 502 shown executes; Figure 3 Steps S301 and S308 shown in FIG can be replaced by Figure 5 The acquisition unit 501 shown in FIG. 5 is executed, and steps S302 to S307 and S309 can be performed by Figure 5 Processing unit 502 is shown executing. Figure 5 The various units in the model training device shown can be separately or all merged into one or several other units to constitute, or a certain (some) unit therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effect of the embodiment of the application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units is realized by one unit. In other embodiments of the present application, the model training device can also include other units. In practical applications, these functions can also be assisted by other units to achieve, and can be achieved by the collaboration of multiple units.

[0207] According to another embodiment of the present application, the following can be executed by running on a general computing device such as a computer device including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM) and other processing elements and storage elements. Figure 2 and Figure 3 A computer program (including program code) for each step involved in the corresponding method shown in Figure 5 The model training device shown in and the model training method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above-mentioned computing device through the computer readable recording medium and run therein.

[0208] Based on the same inventive concept, the principles and beneficial effects of the model training device provided in the embodiment of the present application in solving the problem are similar to the principles and beneficial effects of the model training method in the method embodiment of the present application in solving the problem. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0209] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a terminal device or a server. Figure 6 As shown, the computer device includes at least a processor 601, a communication interface 602, and a memory 603. The processor 601, communication interface 602, and memory 603 may be connected via a bus or other means. The processor 601 (also known as the Central Processing Unit (CPU)) is the computing and control core of the computer device. It can parse various instructions within the computer device and process various data within the computer device. For example, the CPU can be used to parse power on / off commands issued by an object to the computer device and control the computer device to perform power on / off operations. Another example is that the CPU can transmit various interactive data between the internal structures of the computer device, etc. The communication interface 602 may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, a mobile communication interface, etc.), which can be used to send and receive data under the control of the processor 601. The communication interface 602 can also be used for data transmission and interaction within the computer device. The memory 603 is a storage device within the computer device, used to store programs and data. It is understood that the memory 603 here can include both the built-in memory of the computer device and the extended memory supported by the computer device. The memory 603 provides a storage space that stores the operating system of the computer device, which may include but is not limited to: Android system, iOS system, Windows Phone system, etc., and this application does not limit this.

[0210] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, a computer program suitable for being loaded and executed by the processor 601 is also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0211] In one embodiment, the processor 601 performs the following operations by running the computer program in the memory 603:

[0212] Obtaining first mask data and second mask data, where the first mask data and the second mask data are obtained by masking elements in the first object using different masking strategies;

[0213] Performing a first element prediction on the first mask data using an encoder to obtain a first prediction result and a feature vector of the first mask data, wherein the first prediction result includes a prediction result of the masked element in the first mask data;

[0214] Using a decoder to perform second element prediction on the feature vector of the first mask data and the second mask data to obtain a second prediction result, where the second prediction result includes a prediction result of the masked element in the second mask data;

[0215] Based on the first prediction result, the second prediction result and the first object, the encoder and the decoder are jointly trained to obtain a trained encoder, and the trained encoder is used to predict the completion data corresponding to the prompt information.

[0216] As an optional embodiment, a specific embodiment of the processor 601 obtaining the first mask data and the second mask data is as follows:

[0217] Get a first object, where the first object includes M elements, where M is an integer greater than 1;

[0218] Performing random masking on the first object according to a first ratio to obtain first mask data;

[0219] Perform importance perception processing on M elements to obtain the average mutual information of the M elements. The average mutual information of the i-th element is used to indicate the importance of the i-th element in the first object, where i is a positive integer less than or equal to M.

[0220] Key element masking is performed on the first object according to the second ratio and the average mutual information of the M elements to obtain second mask data, where the second ratio is greater than the first ratio.

[0221] As an optional embodiment, the first object is text data, and the elements are characters or words in the text data; the processor 601 performs importance perception processing on M elements to obtain the average mutual information of the M elements. A specific embodiment is as follows:

[0222] Calculate the mutual information of the i-th element based on the number of occurrences of the i-th element in the first object and the number of occurrences of the i-th element's neighboring elements in the first object;

[0223] Perform mean processing on the mutual information of the i-th element to obtain the average mutual information of the i-th element.

[0224] As an optional embodiment, the average mutual information of each element includes an importance score, and the importance score of the i-th element is proportional to the importance of the i-th element in the first object; the processor 601 performs key element masking on the first object according to the second ratio and the average mutual information of the M elements, and obtains the second mask data in a specific embodiment as follows:

[0225] Calculate the ratio of the importance score of each element to the sum of the importance scores of M elements to obtain the mask probability of each element;

[0226] Calculate the number of masked elements of the first object according to the second ratio;

[0227] Based on the number of masked elements of the first object and the masking probabilities of the M elements, masking processing is performed on the first object to obtain second mask data.

[0228] As an optional embodiment, the first object is text data; a specific embodiment of the processor 601 obtaining the first object is:

[0229] Get the data to be processed;

[0230] Normalize the data to be processed to obtain at least one standardized text;

[0231] Identifiers are added to the start and end positions of each specification text to obtain the first object.

[0232] As an optional embodiment, the processor 601 performs normalization processing on the data to be processed to obtain at least one normalized text. A specific embodiment is as follows:

[0233] If there are non-text elements in the data to be processed, the data to be processed is cleaned to obtain the text data corresponding to the data to be processed;

[0234] If the encoding method of the text data is different from the preset encoding method, the encoding method of the text data is converted according to the preset encoding method to obtain text data encoded according to the preset encoding method;

[0235] Based on the text screening rules, the text data encoded according to the preset encoding method is screened to obtain at least one standard text;

[0236] The text screening rule includes at least one of the following: removing duplicate texts with a similarity higher than a similarity threshold, and removing erroneous texts with an error rate higher than an error threshold.

[0237] As an optional embodiment, the second mask data includes a start identifier; the processor 601 uses a decoder to perform a second element prediction on the feature vector of the first mask data and the second mask data, and a specific embodiment of obtaining the second prediction result is:

[0238] Replacing the start identifier of the second mask data with the feature vector of the first mask data to obtain updated second mask data;

[0239] A decoder is used to perform second element prediction on the updated second mask data to obtain a second prediction result.

[0240] As an optional embodiment, the processor 601 jointly trains the encoder and decoder based on the first prediction result, the second prediction result, and the first object, and obtains a specific embodiment of the trained encoder as follows:

[0241] constructing a first loss function based on a difference between a prediction result of a masked element in the first mask data and a corresponding element in the first object;

[0242] constructing a second loss function based on the difference between the prediction results of the masked elements in the second mask data and the corresponding elements in the first object;

[0243] Perform weighted summation on the first loss function and the second loss function to obtain a third loss function;

[0244] The encoder and decoder are jointly trained through the third loss function to obtain the trained encoder.

[0245] As an optional embodiment, the processor 601 further performs the following operations by running the computer program in the memory 603:

[0246] Call the trained encoder to encode N candidate objects to obtain vector representations of N candidate objects, where N is an integer greater than 1;

[0247] Obtain prompt information and call the trained encoder to encode the prompt information to obtain a vector representation of the prompt information;

[0248] Based on the similarity between the vector representation of the prompt information and the vector representation of each candidate object, a second object is selected, wherein the similarity between the vector representation of the second object and the vector representation of the prompt information is greater than a similarity threshold;

[0249] The second object is determined as the completion data corresponding to the prompt information, and the second object is output.

[0250] As an optional embodiment, the processor 601 further performs the following operations by running the computer program in the memory 603:

[0251] Call the trained encoder to encode N candidate objects to obtain vector representations of N candidate objects, each candidate object is associated with a reply message, and N is an integer greater than 1;

[0252] Obtain prompt information and call the trained encoder to encode the prompt information to obtain a vector representation of the prompt information;

[0253] Based on the similarity between the vector representation of the prompt information and the vector representation of each candidate object, a second object is selected, wherein the similarity between the vector representation of the second object and the vector representation of the prompt information is greater than a similarity threshold;

[0254] The reply information associated with the second object is determined as the completion data corresponding to the prompt information, and the reply information associated with the second object is output.

[0255] As an optional embodiment, the encoder includes P neural network structures, and the decoder includes Q neural network structures, where Q is a positive integer and P is an integer greater than Q; each neural network structure includes a multi-head self-attention mechanism layer and a feedforward neural network layer, and the multi-head self-attention mechanism layer and the feedforward neural network layer are respectively associated with a residual connection layer and a normalization layer; the feature vector of the first mask data is the output of the last neural network structure among the P neural network structures.

[0256] Based on the same inventive concept, the principles and beneficial effects of solving problems by the computer device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving problems by the model training method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0257] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the model training method of the above-mentioned method embodiment.

[0258] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described model training method.

[0259] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0260] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0261] In the embodiments of the present application, the "module" or "unit" involved refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0262] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0263] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of this application are still within the scope covered by the application.

Claims

1. A model training method, characterized in that: The method comprises: Obtaining first mask data and second mask data, where the first mask data and the second mask data are obtained by masking elements in the first object using different masking strategies; Performing a first element prediction on the first mask data using an encoder to obtain a first prediction result and a feature vector of the first mask data, where the first prediction result includes a prediction result of the masked element in the first mask data; Using a decoder to perform second element prediction on the feature vector of the first mask data and the second mask data to obtain a second prediction result, where the second prediction result includes a prediction result of the masked element in the second mask data; Based on the first prediction result, the second prediction result and the first object, the encoder and the decoder are jointly trained to obtain a trained encoder, and the trained encoder is used to predict the completion data corresponding to the prompt information.

2. The method according to claim 1, wherein The obtaining of the first mask data and the second mask data includes: Obtain a first object, where the first object includes M elements, where M is an integer greater than 1; Performing random masking processing on the first object according to a first ratio to obtain first mask data; Performing importance perception processing on the M elements to obtain average mutual information of the M elements, where the average mutual information of the i-th element is used to indicate the importance of the i-th element in the first object, where i is a positive integer less than or equal to M; Key element masking is performed on the first object according to a second ratio and the average mutual information of the M elements to obtain second mask data, where the second ratio is greater than the first ratio.

3. The method according to claim 2, wherein The first object is text data, and the elements are characters or words in the text data; performing importance perception processing on the M elements to obtain average mutual information of the M elements includes: Calculating the mutual information of the i-th element based on the number of occurrences of the i-th element in the first object and the number of occurrences of adjacent elements of the i-th element in the first object; Perform mean processing on the mutual information of the i-th element to obtain the average mutual information of the i-th element.

4. The method according to claim 2, wherein The average mutual information of each element includes an importance score, where the importance score of the i-th element is proportional to the importance of the i-th element in the first object; The step of performing key element masking on the first object according to the second ratio and the average mutual information of the M elements to obtain second mask data includes: Calculate the ratio of the importance score of each element to the sum of the importance scores of the M elements to obtain the mask probability of each element; Calculating the number of masked elements of the first object according to the second ratio; Based on the number of masked elements of the first object and the masking probabilities of the M elements, masking processing is performed on the first object to obtain second mask data.

5. The method according to claim 2, wherein The first object is text data, and obtaining the first object includes: Get the data to be processed; Normalizing the data to be processed to obtain at least one standardized text; Identifiers are added to the start and end positions of each specification text to obtain the first object.

6. The method according to claim 5, wherein The step of normalizing the data to be processed to obtain at least one normalized text includes: If there are non-text elements in the data to be processed, the data to be processed is cleaned to obtain text data corresponding to the data to be processed; If the encoding mode of the text data is different from the preset encoding mode, converting the encoding mode of the text data according to the preset encoding mode to obtain text data encoded according to the preset encoding mode; Based on the text screening rules, the text data encoded according to the preset encoding method is screened to obtain at least one standard text; The text screening rule includes at least one of the following: removing duplicate texts with a similarity higher than a similarity threshold, and removing erroneous texts with an error rate higher than an error threshold.

7. The method according to claim 1, wherein The second mask data includes a start identifier; and the decoder is used to perform a second element prediction on the feature vector of the first mask data and the second mask data to obtain a second prediction result, including: Replacing the start identifier of the second mask data with the feature vector of the first mask data to obtain updated second mask data; A decoder is used to perform second element prediction on the updated second mask data to obtain a second prediction result.

8. The method according to claim 1, wherein The step of jointly training the encoder and the decoder based on the first prediction result, the second prediction result, and the first object to obtain a trained encoder includes: constructing a first loss function based on a difference between a prediction result of a masked element in the first mask data and a corresponding element in the first object; constructing a second loss function based on the difference between the prediction result of the masked element in the second mask data and the corresponding element in the first object; Performing weighted summation processing on the first loss function and the second loss function to obtain a third loss function; The encoder and the decoder are jointly trained using the third loss function to obtain a trained encoder.

9. The method according to claim 1, wherein The method further comprises: Calling the trained encoder to encode N candidate objects to obtain vector representations of the N candidate objects, where N is an integer greater than 1; Obtain prompt information, and call the trained encoder to encode the prompt information to obtain a vector representation of the prompt information; Screening out a second object based on the similarity between the vector representation of the prompt information and the vector representation of each candidate object, wherein the similarity between the vector representation of the second object and the vector representation of the prompt information is greater than a similarity threshold; The second object is determined as the complementary data corresponding to the prompt information, and the second object is output.

10. The method according to claim 1, wherein The method further comprises: Calling the trained encoder to encode N candidate objects to obtain vector representations of the N candidate objects, each candidate object is associated with a reply message, and N is an integer greater than 1; Obtain prompt information, and call the trained encoder to encode the prompt information to obtain a vector representation of the prompt information; Screening out a second object based on the similarity between the vector representation of the prompt information and the vector representation of each candidate object, wherein the similarity between the vector representation of the second object and the vector representation of the prompt information is greater than a similarity threshold; The reply information associated with the second object is determined as the completion data corresponding to the prompt information, and the reply information associated with the second object is output.

11. The method according to claim 1, wherein The encoder includes P neural network structures, and the decoder includes Q neural network structures, where Q is a positive integer and P is an integer greater than Q; each neural network structure includes a multi-head self-attention mechanism layer and a feedforward neural network layer, and the multi-head self-attention mechanism layer and the feedforward neural network layer are respectively associated with a residual connection layer and a normalization layer; the feature vector of the first mask data is the output of the last neural network structure arranged among the P neural network structures.

12. A model training device, characterized in that: The model training device comprises: an acquiring unit, configured to acquire first mask data and second mask data, wherein the first mask data and the second mask data are obtained by masking elements in the first object using different masking strategies; a processing unit, configured to perform a first element prediction on the first mask data using an encoder to obtain a first prediction result and a feature vector of the first mask data, wherein the first prediction result includes a prediction result of a masked element in the first mask data; and performing, using a decoder, second element prediction on the feature vector of the first mask data and the second mask data to obtain a second prediction result, wherein the second prediction result includes a prediction result of the masked element in the second mask data; And it is used to jointly train the encoder and the decoder based on the first prediction result, the second prediction result and the first object to obtain a trained encoder, and the trained encoder is used to predict the completion data corresponding to the prompt information.

13. A computer device, characterized in that: include: a memory storing a computer program; A processor, configured to load the computer program to implement the model training method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the model training method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program, which is suitable for being loaded by a processor and executing the model training method according to any one of claims 1 to 11.

Images