Training methods, devices, equipment, and storage media for self-supervised learning models

Abstract

This application discloses a training method, apparatus, device, and storage medium for a self-supervised learning model, belonging to the field of computer and internet technology. The method includes: acquiring a sample set; for a target text sample in the sample set, concatenating the target text sample with other text samples in the sample set to generate a first negative sample corresponding to the target text sample; and using the first negative sample to perform self-supervised training on a text feature extraction model; wherein the text feature extraction model is used to obtain feature information of the input text based on the input text, in order to match retrieval text with semantically similar characteristics to the input text. This application improves the text feature extraction model's ability to distinguish text information with small semantic differences, thereby enhancing the retrieval capability of the text feature extraction model.

Description

Technical Field

[0001] This application relates to the fields of the Internet and computers, and in particular to a training method, apparatus, device and storage medium for a self-supervised learning model. Background Technology

[0002] Currently, during retrieval, the corresponding search information can be obtained by using the retrieval model to retrieve the target information entered by the user.

[0003] In related technologies, during the training of a retrieval model, multiple training samples are obtained from an open-source database and divided into multiple training sample sets, ensuring that no similar training samples exist in a single training sample set. Furthermore, a training sample set is treated as a batch, and the training samples in a batch are input into the retrieval model. The training sample itself is considered a positive sample, and different training samples within a batch are considered negative samples to each other, thereby training the retrieval model.

[0004] However, in the aforementioned related technologies, different training samples within a batch are negative samples of each other, and there are no similar training samples in a batch. That is, there is no similar information in the negative samples of the training samples. After training, the retrieval model can distinguish information that is significantly different, but it cannot distinguish information that is only slightly different. For example, the retrieval model cannot distinguish well between the two pieces of information: "User A cannot log in to the social platform" and "User A logs in to the social platform to express opinions," resulting in poor retrieval performance of the retrieval model. Summary of the Invention

[0005] This application provides a training method, apparatus, device, and storage medium for a self-supervised learning model, which improves the ability of a text feature extraction model to distinguish text information with small semantic differences, thereby enhancing the retrieval capability of the text feature extraction model. The technical solution is as follows.

[0006] According to one aspect of the embodiments of this application, a method for training a self-supervised learning model is provided, the method comprising the following steps:

[0007] Obtain a sample set, which includes at least two text samples;

[0008] For the target text sample in the sample set, the target text sample and other text samples in the sample set are concatenated to generate the first negative sample corresponding to the target text sample.

[0009] The text feature extraction model is trained in a self-supervised manner using the first negative sample corresponding to the target text sample; wherein, the text feature extraction model is used to obtain the feature information of the input text based on the input text, so as to match the retrieval text that is semantically similar to the input text.

[0010] According to one aspect of the embodiments of this application, a training apparatus for a self-supervised learning model is provided, the apparatus comprising the following modules:

[0011] A set acquisition module is used to acquire a sample set, wherein the sample set includes at least two text samples;

[0012] The sample generation module is used to concatenate the target text sample and other text samples in the sample set, except for the target text sample, to generate a first negative sample corresponding to the target text sample.

[0013] The model training module is used to perform self-supervised training on the text feature extraction model using the first negative sample corresponding to the target text sample; wherein, the text feature extraction model is used to obtain the feature information of the input text based on the input text in order to match the retrieval text that is semantically similar to the input text.

[0014] According to one aspect of the embodiments of this application, an embodiment of this application provides a computer device, the computer device including a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the training method of the self-supervised learning model described above.

[0015] According to one aspect of the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing at least one program, which is loaded and executed by a processor to implement the training method of the self-supervised learning model described above.

[0016] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method for the self-supervised learning model described above.

[0017] The technical solution provided in this application can bring the following beneficial effects:

[0018] By concatenating a text sample with other text samples in the sample set, a first negative sample corresponding to the text sample is obtained. This first negative sample includes all or part of the information of the text sample, reducing the semantic gap between the first negative sample and the text sample. Subsequently, the text feature extraction model is self-supervised based on the first negative sample, which improves the ability of the text feature extraction model to distinguish text information with small semantic differences. This makes the retrieval information based on the input information determined by the text feature extraction model more accurate, thereby improving the retrieval capability of the text feature extraction model. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the training method of a self-supervised learning model provided in one embodiment of this application;

[0020] Figure 2 An exemplary schematic diagram of a text feature extraction system is shown;

[0021] Figure 3 This is a flowchart of a training method for a self-supervised learning model provided in one embodiment of this application;

[0022] Figure 4 An exemplary diagram illustrates the training and usage process of a text feature extraction model;

[0023] Figure 5 This is a block diagram of a training apparatus for a self-supervised learning model provided in one embodiment of this application;

[0024] Figure 6 This is a block diagram of a training apparatus for a self-supervised learning model provided in another embodiment of this application;

[0025] Figure 7 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0027] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0028] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0029] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0030] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0031] The solution provided in this application relates to artificial intelligence technologies such as machine learning. It generates a first negative sample corresponding to a text sample by concatenating and inserting text samples, and then uses this first negative sample to train a text feature extraction model. This first negative sample can also be referred to as a hard negative sample. For example, as... Figure 1As shown, for a text sample in the sample set, a positive sample corresponding to the text sample is generated based on the text sample, and other text samples in the sample set except for the text sample are concatenated and inserted into the text sample to generate a hard negative sample corresponding to the text sample. Further, for the text feature extraction model, the neurons of each layer of the network in the text feature extraction model are randomly masked to obtain three different mask models, and the model parameters masked by the different mask models are different. Then, the above text sample is input into the first mask model to obtain sample feature information, the above positive sample is input into the second mask model to obtain positive sample feature information, and the above hard negative sample is input into the third mask model to obtain hard negative sample feature information. Then, the text feature extraction model is self-supervised trained based on the sample feature information, positive sample feature information, and hard negative sample feature information.

[0032] Please refer to Figure 2 This illustration shows a schematic diagram of a text feature extraction system provided in one embodiment of this application. The text feature extraction system may include a terminal device 10 and a server 20.

[0033] Terminal device 10 can be electronic devices such as mobile phones, tablets, PCs (Personal Computers), intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc., and this application embodiment does not limit it.

[0034] Server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0035] The aforementioned terminal device 10 and the aforementioned server 20 can be connected directly or indirectly through wired or wireless communication, and this application does not impose any restrictions on this.

[0036] In some embodiments, the server 20 includes a text feature extraction model, and the terminal device 10 includes any application with retrieval functionality. For example... Figure 2 As shown, the terminal device 10 obtains the input text entered by the user based on the user interface of the application and sends the input text to the server 20; correspondingly, the server 20 inputs the input text into the text feature extraction model to obtain the feature information corresponding to the input text, and matches the semantically similar search text for the input text based on the feature information, and sends the semantically similar search text to the terminal device 10, so that the terminal device 10 displays the search text to the user in the user interface.

[0037] It should be noted that the above text feature extraction model can be trained on server 20 or on other computer devices. For example... Figure 2 As shown, the text feature extraction system includes a training system for a self-supervised learning model. This training system includes at least one model training device 30, which trains the text feature extraction model and provides the trained model to the server 20. In some embodiments, the server 20 also serves as the model training device 30 within the self-supervised learning model training system.

[0038] Please refer to Figure 3 This document illustrates a flowchart of a training method for a self-supervised learning model provided in an embodiment of this application. The steps in this method can be performed by the methods described above. Figure 2 The method is executed by the server 20 / model training device 30 (hereinafter collectively referred to as the "computer device"). The method may include at least one of the following steps (301-303):

[0039] Step 301: Obtain the sample set.

[0040] A sample set refers to the set of training samples for a text feature extraction model, and this sample set contains at least two text samples. In this embodiment, the computer device acquires the sample set before training the text feature extraction model.

[0041] In some embodiments, the sample set can be pre-stored information or information acquired in real time. In one possible implementation, the sample set is pre-stored information. In some embodiments, after determining that the text feature extraction model has been trained, the computer device acquires the sample set from the corresponding storage location based on the storage location of the sample set. The storage location can be located on the computer device itself or on other devices besides the computer device; this application embodiment does not limit this. In another possible implementation, the sample set is information acquired in real time. In some embodiments, after determining that the text feature extraction model has been trained, the computer device collects and acquires the sample set in real time from the network environment. The computer device can rely on its own device to collect and acquire the sample set or rely on other devices besides the computer device to collect and acquire the sample set; this application embodiment does not limit this.

[0042] In some embodiments, to improve the training performance of the text feature extraction model, the sample set does not include semantically similar text samples. For example, after obtaining multiple text samples, semantic clustering is performed on the text samples, and then semantically similar text samples are classified into different sample sets based on the clustering results, so that the same sample set does not include semantically similar text samples.

[0043] Step 302: For the target text sample in the sample set, concatenate the target text sample and other text samples in the sample set to generate the first negative sample corresponding to the target text sample.

[0044] In this embodiment of the application, after acquiring the aforementioned sample set, the computer device concatenates the text sample and other text samples in the sample set to generate a first negative sample corresponding to the text sample. Taking a target text sample as an example, for the target text sample in the sample set, the target text sample and other text samples in the sample set are concatenated to generate a first negative sample corresponding to the target text sample.

[0045] The aforementioned target text sample refers to any text sample in the sample set, and the aforementioned first negative sample refers to the hard negative sample. The target text sample corresponds to the first negative sample, which includes the key text information of the target text sample's semantics; that is, the key text information of the first negative sample includes the key text information of the target text sample.

[0046] In some embodiments, the first negative sample may include all the key text information in the target text sample, or the first negative sample may also include some of the key text information in the target text sample; this application embodiment does not limit this. For example, the key text information of the target text sample includes key text information a and key text information b, and the key text information of the first negative sample includes key text information a, key text information b, and key text information c; or, the key text information of the target text sample includes key text information a and key text information b, and the key text information of the first negative sample includes key text information a and key text information c.

[0047] In one possible implementation, the first negative sample includes all the key text information of the target text sample, that is, during the above-mentioned splicing process, all the key text information of the target text sample is spliced ​​and inserted into the other text samples. In some embodiments, for the target text sample, the computer device splices and inserts the other text samples into the target text sample to ensure that all the key text information of the target text sample is spliced ​​and inserted into the other text samples. For example, if the target text sample is "User A expresses his opinion on the first event" and the other text sample is "User B responds to the second event", then the first negative sample is "User A expresses his opinion on the first event and User B responds to the second event"; or, the computer device obtains at least one key text information of the target text sample from the target text sample, and then splices and inserts all the key text information of the target text sample into the other text samples.

[0048] In another possible implementation, the first negative sample includes some key text information of the target text sample. That is, during the above-mentioned splicing process, some key text information of the target text sample is spliced ​​and inserted into the other text samples. In some embodiments, for a target text sample, the computer device obtains at least one key text information of the target text sample from the target text sample, and then selects target key text information from the at least one key text information to splice and insert into at least the other text samples. The target key text information can be any one of the at least one key text information. For example, one or more key text information are randomly selected from the at least one key text information as the target key text information; or, the target key text information can be key text information that is semantically similar to other text samples from the at least one key text information. For example, one or more key text information whose semantic distance from other text samples is less than a distance threshold are selected from the at least one key text information as the target key text information. The distance threshold can be any data, and can be flexibly set and adjusted according to actual conditions. This application embodiment does not limit this. For example, the distance threshold can be a pre-set threshold; or it can be a threshold determined based on one or more minimum semantic distances after obtaining the semantic distance between the key text information and other text samples.

[0049] In some embodiments, the computer device obtains the aforementioned key text information through word segmentation. Taking a target text sample as an example, after obtaining the target text sample, the computer device performs word segmentation on the target text sample to obtain at least one word, and then, based on the part of speech of each word, uses verbs and / or nouns as key text information of the target text sample. Here, the aforementioned nouns refer to nouns located after and adjacent to the verb.

[0050] It should be noted that the number of the other text samples mentioned above can be any value, and this value can be flexibly set and adjusted according to the actual situation. This application embodiment does not limit this.

[0051] Step 303: Use the first negative sample corresponding to the target text sample to perform self-supervised training on the text feature extraction model.

[0052] A text feature extraction model is used to obtain feature information of the input text based on the input text, in order to match retrieval text that is semantically similar to the input text. In this embodiment, after acquiring the first negative sample, the computer device uses the first negative sample corresponding to the target text sample to perform self-supervised training on the text feature extraction model.

[0053] In some embodiments, the target text sample corresponds to a positive sample. When training the text feature extraction model, the computer device uses the target text sample, the positive sample corresponding to the target text sample, and the first negative sample corresponding to the target text sample for self-supervised training. In one possible implementation, the positive sample corresponding to the target text sample is the target text sample itself. In another possible approach, the positive sample corresponding to the target text sample is generated by replacing key text information; for example, after obtaining the target text sample, the computer device obtains at least one key text information from the target text sample, and then performs synonym replacement on all or part of the key text information to generate the positive sample corresponding to the target text sample.

[0054] In some embodiments, the target text sample also corresponds to a second negative sample. This second negative sample refers to a easily negative sample; in the sample set, different text samples are each other's second negative samples. In one possible implementation, during the training of the text feature extraction model, the target text sample, the corresponding positive sample, the corresponding first negative sample, and the corresponding second negative sample are used to perform self-supervised training on the text feature extraction model. In another possible implementation, during the training of the text feature extraction model, the target text sample, the corresponding positive sample, and the corresponding second negative sample are first used to perform self-supervised training on the initial text feature extraction model to obtain the aforementioned text feature extraction model. Further, the target text sample, the corresponding positive sample, and the corresponding first negative sample are used to continue self-supervised training on the text feature extraction model to achieve model optimization.

[0055] In summary, the technical solution provided in this application involves concatenating a text sample with other text samples in the sample set to obtain a first negative sample corresponding to the text sample. This first negative sample includes all or part of the information of the text sample, reducing the semantic gap between the first negative sample and the text sample. Subsequently, the text feature extraction model is trained under self-supervised conditions based on the first negative sample, which improves the ability of the text feature extraction model to distinguish text information with small semantic differences. This makes the retrieval information based on the input information determined by the text feature extraction model more accurate, thereby improving the retrieval capability of the text feature extraction model.

[0056] In addition, this application provides an automatic generation scheme for the first negative sample. For text samples in the sample set, the first negative sample can be guaranteed to include all or part of the information of the text samples by concatenating the text samples with other text samples. Compared with the manual annotation of the first negative sample in related technologies, or the selection of samples with small semantic distance from the sample database as the first negative sample in related technologies, the first negative sample can be generated based on the information contained in the sample set itself, which improves the generation efficiency of the first negative sample, is simple to operate, and reduces the consumption of human resources.

[0057] The generation method of the first negative sample will be described below.

[0058] In an exemplary embodiment, step 302 above includes at least one of the following:

[0059] 1. From the other text samples in the sample set besides the target text sample, determine the interference text sample corresponding to the target text sample.

[0060] Interference text samples refer to text samples that interfere with the semantics of target text samples. In the embodiments of this application, after acquiring the target text sample value, the computer device determines the interference sample corresponding to the target text sample from other text samples in the sample set besides the target text sample.

[0061] In one possible implementation, the aforementioned interfering text sample refers to any other text sample in the sample set. In some embodiments, the computer device randomly selects another text sample from the other text samples in the sample set as the aforementioned interfering text sample.

[0062] In another possible implementation, the aforementioned interfering text samples refer to other text samples in the sample set whose semantic distance from the target text sample satisfies the first condition. In some embodiments, the computer device acquires the semantic distance between each other text sample and the target text sample; further, other text samples with a semantic distance less than a first threshold are determined as interfering text samples corresponding to the target text sample. In this case, selecting other text samples with small semantic distances as interfering text samples increases the similarity between the interfering text samples and the target text sample, reduces the semantic gap between the first negative sample and the text sample, and enables the text feature extraction model to improve its ability to distinguish text information with small semantic differences after self-supervised training based on the first negative sample. The first threshold can be any value, and can be flexibly set and adjusted according to actual conditions; this application embodiment does not limit this. For example, the first threshold can be a pre-set threshold; or it can be a threshold determined based on one or more of the smallest semantic distances after acquiring the semantic distances between other text samples and the target text sample.

[0063] In some embodiments, when obtaining the semantic distance between other text samples and the target text sample, the computer device obtains the semantic distance between the other text samples and the target text sample based on the aforementioned text feature extraction model. This text feature extraction model is trained based on a first negative sample corresponding to the target text sample. The first negative sample refers to other text samples in the sample set besides the target text sample, and different text samples in the sample set are each other's second negative samples. For example, the computer device inputs each text sample in the sample set into the text feature extraction model to obtain feature information corresponding to each text sample; further, based on the feature information corresponding to each text sample, it obtains the semantic distance between each other text sample and the target text sample. Of course, in other possible embodiments, the computer device may also use other methods to obtain the semantic distance. For example, the computer device determines the semantic distance based on the length of similar strings between other text samples and the target text sample; or, the computer device determines the semantic distance based on the corpus similarity between other text samples and the target text sample; or, the computer device determines the semantic distance based on the knowledge similarity between other text samples and the target text sample; and so on.

[0064] 2. Insert interfering text samples into the target text sample to generate the first negative sample corresponding to the target text sample.

[0065] In this embodiment of the application, after acquiring the aforementioned interfering text sample, the computer device splices and inserts the interfering text sample into the target text sample to generate the first negative sample corresponding to the target text sample.

[0066] In some embodiments, the computer device inserts the entire text of the interfering text sample into the target text sample. The insertion position of the interfering text sample into the target text sample can be random or at a specific position; this embodiment does not limit this.

[0067] In one possible implementation, the insertion position of the interfering text sample relative to the target text sample is random. In some embodiments, the computer device randomly determines the insertion position of the interfering text sample from the target text sample, and then splices and inserts the interfering text sample at that insertion position.

[0068] In another possible implementation, the insertion position of the interfering text sample relative to the target text sample is a specific location. In some embodiments, the computer device segments the target text sample to obtain at least one text fragment, and obtains the semantic distance between each text fragment and the interfering text sample; further, based on the position of the text fragment with a semantic distance less than a second threshold in the target text sample, the interfering text sample is concatenated and inserted into the target text sample to generate a first negative sample corresponding to the target text sample. In this case, concatenating and inserting the interfering text sample before or after the text fragment with a small semantic distance reduces the semantic gap between the first negative sample and the text sample, thereby improving the ability of the text feature extraction model to distinguish text information with small semantic differences when subsequently self-supervised training is performed based on the first negative sample.

[0069] For example, in the above segmentation process, the number of text segments is n, and n is a positive integer; the computer device segments the target text sample into at least one text sub-segment, and the different text sub-segments do not include duplicate text; further, based on the arrangement order of each text sub-segment in the target text sample, the first i text sub-segments are sequentially concatenated to generate the i-th text segment, and i is a positive integer less than or equal to n.

[0070] For example, regarding the semantic distance between the aforementioned text fragment and the interfering text sample, the computer device can obtain the semantic distance between the text fragment and the interfering text sample based on the aforementioned text feature extraction model, or it can obtain the semantic distance between the text fragment and the interfering text sample based on similar string length, corpus similarity, knowledge similarity, etc. The embodiments of this application do not limit this.

[0071] For example, the second threshold can be any value, and can be flexibly set and adjusted according to the actual situation. This application embodiment does not limit this. For example, the second threshold can be a preset threshold; or it can be a threshold determined based on the minimum semantic distance after obtaining the semantic distance between the text fragment and the interfering text sample.

[0072] For example, regarding the above-mentioned splicing insertion, the interfering text sample can be spliced ​​and inserted before or after the text segment whose semantic distance is less than the second threshold. This application embodiment does not limit this. It should be noted that in the first negative sample, the interfering text sample is adjacent to the text segment whose semantic distance is less than the second threshold.

[0073] Of course, in other possible implementations, the staff can pre-set the above-mentioned insertion position, and then directly insert the text based on that position. For example, the insertion position can be: the beginning of the target text sample, the end of the target text sample, the middle of the target text sample, or after the second character of the target text sample, etc. This application embodiment does not limit this.

[0074] In some embodiments, the computer device concatenates and inserts key text information from interfering text samples into the target text sample. For example, the computer device obtains key text information from the interfering text samples; further, it concatenates and inserts the key text information into the target text sample to generate a first negative sample corresponding to the target text sample. In this case, concatenating and inserting the key text information from the interfering text samples into the target text sample avoids the training burden caused by excessive information in the first negative sample, thus improving the training efficiency of the text feature extraction model.

[0075] The key text information of the interfering text sample can be inserted at a random or a specific position relative to the target text sample. This application does not limit this.

[0076] In one possible implementation, the insertion position of the key text information of the interfering text sample relative to the target text sample is random. In some embodiments, the computer device randomly determines the insertion position of the key text information of the interfering text sample from the target text sample, and then splices and inserts the key text information of the interfering text sample at that insertion position.

[0077] In another possible implementation, the key text information of the interfering text sample is inserted at a specific position relative to the target text sample. In some embodiments, the computer device segments the target text sample to obtain at least one text fragment, and obtains the semantic distance between each text fragment and the key text information of the interfering text sample; further, based on the position of the text fragment with a semantic distance less than a third threshold in the target text sample, the key text information of the interfering text sample is concatenated and inserted into the target text sample to generate a first negative sample corresponding to the target text sample.

[0078] For example, the above segmentation process is described in detail above and will not be repeated here.

[0079] For example, regarding the semantic distance between the key text information of the above-mentioned text fragment and the interfering text sample, the computer device can obtain the semantic distance between the key text information of the text fragment and the interfering text sample according to the above-mentioned text feature extraction model, or it can obtain the semantic distance between the key text information of the text fragment and the interfering text sample according to similar string length, corpus similarity, knowledge similarity, etc. The embodiments of this application do not limit this.

[0080] For example, the aforementioned third threshold can be any value, and can be flexibly set and adjusted according to actual conditions. This application embodiment does not limit this. For example, the third threshold can be a preset threshold; or it can be a threshold determined based on the minimum semantic distance after obtaining the semantic distance between the key text information of the text fragment and the interfering text sample.

[0081] For example, regarding the above-mentioned splicing insertion, the key text information of the interfering text sample can be spliced ​​and inserted before or after the text segment whose semantic distance is less than the third threshold. This application embodiment does not limit this. It should be noted that in the first negative sample, the key text information of the interfering text sample is adjacent to the text segment whose semantic distance is less than the third threshold.

[0082] Of course, in other possible implementations, the staff can pre-set the above insertion position, and then directly insert and splice according to the insertion position.

[0083] It should be noted that the key text information of the aforementioned interference text sample can be all the key text information of the interference text sample, or it can be a portion of the key text information of the interference text sample; this application embodiment does not limit this. Furthermore, the method of obtaining the key text information of the interference text sample is related to... Figure 3 The method for obtaining key text information of the target text sample in the embodiments is similar; see details below. Figure 3 Examples are not described in detail here.

[0084] In summary, the technical solution provided in this application provides an automatic method for generating first negative samples by splicing and inserting interfering text samples into text samples. This method is simple to operate, reduces human resource consumption, and improves the generation efficiency of first negative samples.

[0085] The training method for the above text feature extraction model will be introduced below.

[0086] In an exemplary embodiment, step 303 above includes at least one of the following:

[0087] 1. Based on the text feature extraction model, generate the first mask model, the second mask model, and the third mask model.

[0088] In this embodiment, before training the text feature extraction model, the computer device generates a first mask model, a second mask model, and a third mask model based on the text feature extraction model. The mask model refers to the model obtained by masking some model parameters of each layer of the original model. Due to the randomness of the mask parameters, different mask models correspond to different model parameters.

[0089] In some embodiments, a computer device randomly masks the model parameters of each layer of the text feature extraction model to obtain a first mask model, a second mask model, and a third mask model. Different mask models mask different model parameters; therefore, different mask models correspond to different model parameters.

[0090] In one possible implementation, model parameters are masked by discarding neurons. For example, a text feature extraction model includes a multi-layer network, with each layer containing multiple neurons. For a target network within the multi-layer network, one or more neurons in that target network are randomly discarded to mask its parameters. It should be noted that the number of neurons discarded in different networks can be the same or different; this embodiment does not limit this.

[0091] 2. Input the target text sample, the corresponding positive sample of the target text sample, and the first negative sample corresponding to the target text sample into different mask models.

[0092] In this embodiment of the application, after acquiring the first mask model, the second mask model, and the third mask model, the computer device uses the first mask model as a mask model for the target text sample, and then inputs the target text sample into the first mask model to obtain sample feature information; and uses the second mask model as a mask model for positive samples, and inputs the positive sample corresponding to the target text sample into the second mask model to obtain positive sample feature information; and uses the third mask model as a mask model for the first negative sample, and inputs the first negative sample corresponding to the target text sample into the third mask model to obtain first negative sample feature information.

[0093] 3. Based on the sample feature information, positive sample feature information, and first negative sample feature information, determine the first semantic distance and second semantic distance corresponding to the target text sample.

[0094] In this embodiment of the application, after acquiring the above-mentioned sample feature information, positive sample feature information and first negative sample feature information, the computer device determines the first semantic distance corresponding to the target text sample based on the sample feature information and positive sample feature information; and determines the second semantic distance corresponding to the target text sample based on the sample feature information and first negative sample feature information.

[0095] 4. The text feature extraction model is trained in a self-supervised manner based on the first semantic distance and the second semantic distance corresponding to each text sample.

[0096] In this embodiment, after obtaining the first semantic distance and the second semantic distance, the computer device performs self-supervised training on the text feature extraction model based on the first semantic distance and the second semantic distance corresponding to each text sample. It should be noted that the self-supervised training of the text feature extraction model involves multiple iterations. For example, after a certain iteration of training, if the text feature extraction model is not yet fully trained, the steps of generating the first mask model, the second mask model, and the third mask model based on the text feature extraction model are restarted until the text feature extraction model is fully trained.

[0097] In some embodiments, the target text sample described above also corresponds to a second negative sample. This second negative sample refers to a sample prone to negativeness.

[0098] In one possible implementation, during the training of the text feature extraction model, the target text sample, the corresponding positive sample, the corresponding first negative sample, and the corresponding second negative sample are used to perform self-supervised training on the text feature extraction model. In some embodiments, the computer device generates a fourth mask model based on the above text feature extraction model, and inputs the corresponding second negative sample of the target text sample into the fourth mask model to obtain the second sample feature information; further, based on the sample feature information and the second negative sample feature information, the third semantic distance corresponding to the target text sample is determined, and the text feature extraction model is self-supervised trained based on the first semantic distance, second semantic distance, and third semantic distance corresponding to each text sample, respectively. It should be noted that after a certain iteration of training, if the text feature extraction model is not fully trained, four new mask models are obtained and the above steps are repeated until the text feature extraction model is fully trained.

[0099] In another possible implementation, during the training of the text feature extraction model, the initial text feature extraction model is first trained in a self-supervised manner using the target text sample, the corresponding positive sample, and the corresponding second negative sample to obtain the aforementioned text feature extraction model. Further, the target text sample, the corresponding positive sample, and the corresponding first negative sample are used to continue the self-supervised training of the text feature extraction model. In some embodiments, the computer device determines other text samples in the sample set besides the target text sample as the corresponding second negative sample. Further, the corresponding second negative sample is used to train the initial text feature extraction model in a self-supervised manner to obtain the aforementioned text feature extraction model. Then, the target text sample, the corresponding positive sample, and the corresponding first negative sample are used to continue the self-supervised training of the text feature extraction model. It should be noted that the self-supervised training of the initial text feature extraction model can be performed using a mask model as described above, or the initial text feature extraction model can be trained directly; this application embodiment does not limit this approach.

[0100] In summary, the technical solution provided in this application generates different mask models by masking model parameters. Furthermore, text samples, positive samples, and negative samples are input into different mask models respectively. The implicit structure of the model is modified in each training iteration, which can effectively prevent overfitting problems in the text feature extraction model. Moreover, it can generate diverse positive samples and diverse first negative samples in each training iteration, thereby improving the training effect of the text feature extraction model.

[0101] Furthermore, in the embodiments of this application, the above-mentioned text feature extraction model can also be referred to as a retrieval model. Below, as follows... Figure 4 As shown, taking a retrieval model as an example, a complete introduction to the training and use of the model is provided. The specific steps are as follows:

[0102] During model training:

[0103] Step 401: Obtain a sample set, which includes at least two text samples.

[0104] Step 402: For the target text sample in the sample set, other text samples in the sample set, excluding the target text sample, are concatenated and inserted into the target text sample to generate the first negative sample corresponding to the target text sample.

[0105] Step 403: Generate positive samples corresponding to the target text samples based on the target text samples.

[0106] Step 404: Train the retrieval model using the target text sample, the positive sample corresponding to the target text sample, and the first negative sample corresponding to the target text sample.

[0107] During the offline process after model training is complete:

[0108] Step 405: Obtain multiple search texts from the search text database. The number of search texts can be arbitrary, and this embodiment does not limit this.

[0109] Step 406: Input each search text into the search model to obtain the feature information corresponding to each search text.

[0110] During the online process after model training is complete:

[0111] Step 407: Obtain the input text entered by the user account. In some embodiments, the computer device obtains the input text through a client.

[0112] Step 408: Input the input text into the retrieval model to obtain the feature information corresponding to the input text.

[0113] Step 409: Based on the feature information corresponding to the input text and the feature information corresponding to each search text, obtain the semantic distance between each search text and the input text.

[0114] Step 410: The search texts with a semantic distance less than the target value are determined as the search texts corresponding to the input text. The target value can be any data, and can be flexibly set and adjusted according to actual circumstances; this embodiment does not limit this.

[0115] In some embodiments, if the input text is obtained through a client, the computer device can send the search text corresponding to the input text to the client, enabling the client to display the corresponding search text to the user, or to display information related to the search text. The information related to the search text may be text with the search text as its title, or text with the search text as its key text information, etc., and this application embodiment does not limit this.

[0116] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0117] Please refer to Figure 5 This diagram illustrates a block diagram of a training apparatus for a self-supervised learning model according to an embodiment of this application. The apparatus has the function of implementing the training method for the self-supervised learning model described above. This function can be implemented in hardware or by hardware executing corresponding software. The apparatus 500 can be a computer device or be installed within a computer device. The apparatus 500 may include: a set acquisition module 510, a sample generation module 520, and a model training module 530.

[0118] The set acquisition module 510 is used to acquire a sample set, which includes at least two text samples.

[0119] The sample generation module 520 is used to concatenate the target text sample and other text samples in the sample set, except for the target text sample, to generate a first negative sample corresponding to the target text sample.

[0120] The model training module 530 is used to perform self-supervised training on the text feature extraction model using the first negative sample corresponding to the target text sample; wherein, the text feature extraction model is used to obtain the feature information of the input text based on the input text in order to match the retrieval text that is semantically similar to the input text.

[0121] In an exemplary embodiment, such as Figure 6 As shown, the sample generation module 520 includes a sample determination unit 521 and a sample generation unit 522.

[0122] The sample determination unit 521 is used to determine the interference text sample corresponding to the target text sample from other text samples in the sample set besides the target text sample.

[0123] The sample generation unit 522 is used to concatenate and insert the interference text sample into the target text sample to generate the first negative sample corresponding to the target text sample.

[0124] In an exemplary embodiment, the sample determination module 521 is further configured to:

[0125] The semantic distance between each of the other text samples and the target text sample is obtained respectively;

[0126] Other text samples whose semantic distance is less than the first threshold are identified as interference text samples corresponding to the target text sample.

[0127] In an exemplary embodiment, the sample generation unit 522 is further configured to:

[0128] The target text sample is segmented to obtain at least one text fragment;

[0129] The semantic distance between each of the text segments and the interfering text samples is obtained respectively;

[0130] Based on the position of text fragments with a semantic distance less than a second threshold in the target text sample, the interfering text samples are spliced ​​and inserted into the target text sample to generate the first negative sample corresponding to the target text sample.

[0131] In an exemplary embodiment, the sample generation unit 522 is further configured to:

[0132] Extract key text information from the interference text samples;

[0133] The key text information is concatenated and inserted into the target text sample to generate the first negative sample corresponding to the target text sample.

[0134] In an exemplary embodiment, such as Figure 6 As shown, the model training module 530 includes: a model masking unit 531, a feature extraction unit 532, a distance determination unit 533, and a model training unit 534.

[0135] The model masking unit 531 is used to generate a first masking model, a second masking model, and a third masking model based on the text feature extraction model; wherein, different masking models correspond to different model parameters.

[0136] The feature extraction unit 532 is used to input the target text sample into the first mask model to obtain sample feature information; input the positive sample corresponding to the target text sample into the second mask model to obtain positive sample feature information; and input the first negative sample corresponding to the target text sample into the third mask model to obtain first negative sample feature information.

[0137] The distance determination unit 533 is used to determine the first semantic distance corresponding to the target text sample based on the sample feature information and the positive sample feature information; and to determine the second semantic distance corresponding to the target text sample based on the sample feature information and the first negative sample feature information.

[0138] The model training unit 534 is used to perform self-supervised training on the text feature extraction model based on the first semantic distance and the second semantic distance corresponding to each of the text samples.

[0139] In an exemplary embodiment, the model masking unit 531 is further configured to randomly mask the model parameters of each layer of the text feature extraction model to obtain the first mask model, the second mask model, and the third mask model; wherein the model parameters masked by different mask models are different.

[0140] In summary, the technical solution provided in this application involves concatenating a text sample with other text samples in the sample set to obtain a first negative sample corresponding to the text sample. This first negative sample includes all or part of the information of the text sample, reducing the semantic gap between the first negative sample and the text sample. Subsequently, the text feature extraction model is trained under self-supervised conditions based on the first negative sample, which improves the ability of the text feature extraction model to distinguish text information with small semantic differences. This makes the retrieval information based on the input information determined by the text feature extraction model more accurate, thereby improving the retrieval capability of the text feature extraction model.

[0141] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0142] Please refer to Figure 7 This diagram illustrates the structural block diagram of a computer device provided in one embodiment of this application. This computer device can be used to implement the training method for the self-supervised learning model described above. Specifically:

[0143] Computer device 700 includes a central processing unit (CPU) 701, a system memory 704 including random access memory (RAM) 702 and read-only memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the CPU 701. Computer device 700 also includes a basic input / output system (I / O system) 706 that facilitates information transfer between various devices within the computer, and a mass storage device 707 for storing the operating system 713, application programs 714, and other program modules 715.

[0144] The basic input / output system 706 includes a display 708 for displaying information and an input device 709 for user input, such as a mouse or keyboard. Both the display 708 and the input device 709 are connected to the central processing unit 701 via an input / output controller 710 connected to the system bus 705. The basic input / output system 706 may also include the input / output controller 710 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 710 also provides output to a display screen, printer, or other types of output devices.

[0145] Mass storage device 707 is connected to central processing unit 701 via a mass storage controller (not shown) connected to system bus 705. Mass storage device 707 and its associated computer-readable media provide non-volatile storage for computer device 700. That is, mass storage device 707 may include computer-readable media (not shown) such as hard disk or CD-ROM (CompactDisc Read-Only Memory) drive.

[0146] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage devices, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 704 and mass storage device 707 described above can be collectively referred to as memory.

[0147] According to various embodiments of this application, the computer device 700 can also be connected to a remote computer on a network, such as the Internet, for operation. That is, the computer device 700 can be connected to a network 712 via a network interface unit 711 connected to the system bus 705, or the network interface unit 711 can be used to connect to other types of networks or remote computer systems (not shown).

[0148] The memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the training method of the self-supervised learning model described above.

[0149] In an exemplary embodiment, a computer-readable storage medium is also provided, the storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set, when executed by a processor, implements the training method of the self-supervised learning model described above.

[0150] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0151] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method for the self-supervised learning model described above.

[0152] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0153] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a self-supervised learning model, characterized in that, The method includes: Obtain a sample set, which includes at least two text samples; For the target text sample in the sample set, the target text sample and other text samples in the sample set are concatenated to generate the first negative sample corresponding to the target text sample. Based on the text feature extraction model, a first mask model, a second mask model, and a third mask model are generated. The target text sample is input into the first mask model to obtain sample feature information. The positive sample corresponding to the target text sample is input into the second mask model to obtain positive sample feature information. The first negative sample corresponding to the target text sample is input into the third mask model to obtain first negative sample feature information. Based on the sample feature information and the positive sample feature information, a first semantic distance corresponding to the target text sample is determined. Based on the sample feature information and the first negative sample feature information, a second semantic distance corresponding to the target text sample is determined. Based on the first semantic distance and the second semantic distance corresponding to each text sample, the text feature extraction model is self-supervised trained. The text feature extraction model is used to obtain the feature information of the input text based on the input text to match retrieval text that is semantically similar to the input text. Different mask models correspond to different model parameters. The positive sample is the target text sample; or, the positive sample is generated by replacing key text information of the target text sample.

2. The method according to claim 1, characterized in that, The step of concatenating the target text sample and other text samples in the sample set (excluding the target text sample) to generate the first negative sample corresponding to the target text sample includes: From the other text samples in the sample set besides the target text sample, determine the interference text sample corresponding to the target text sample; The interference text sample is concatenated and inserted into the target text sample to generate the first negative sample corresponding to the target text sample.

3. The method according to claim 2, characterized in that, Determining the interference text sample corresponding to the target text sample from other text samples in the sample set besides the target text sample includes: The semantic distance between each of the other text samples and the target text sample is obtained respectively; Other text samples whose semantic distance is less than the first threshold are identified as interference text samples corresponding to the target text sample.

4. The method according to claim 2, characterized in that, The step of concatenating and inserting the interfering text sample into the target text sample to generate the first negative sample corresponding to the target text sample includes: The target text sample is segmented to obtain at least one text fragment; The semantic distance between each of the text segments and the interfering text samples is obtained respectively; Based on the position of text fragments with a semantic distance less than a second threshold in the target text sample, the interfering text samples are spliced ​​and inserted into the target text sample to generate the first negative sample corresponding to the target text sample.

5. The method according to claim 2, characterized in that, The step of concatenating and inserting the interfering text sample into the target text sample to generate the first negative sample corresponding to the target text sample includes: Extract key text information from the interference text samples; The key text information is concatenated and inserted into the target text sample to generate the first negative sample corresponding to the target text sample.

6. The method according to any one of claims 1 to 5, characterized in that, The text feature extraction model generates a first mask model, a second mask model, and a third mask model, including: The model parameters of each layer of the text feature extraction model are randomly masked to obtain the first mask model, the second mask model, and the third mask model; Different masking models mask different model parameters.

7. A training device for a self-supervised learning model, characterized in that, The device includes: A set acquisition module is used to acquire a sample set, wherein the sample set includes at least two text samples; The sample generation module is used to concatenate the target text sample and other text samples in the sample set, except for the target text sample, to generate a first negative sample corresponding to the target text sample. The model training module is used to generate a first mask model, a second mask model, and a third mask model based on a text feature extraction model; input the target text sample into the first mask model to obtain sample feature information; input the positive sample corresponding to the target text sample into the second mask model to obtain positive sample feature information; input the first negative sample corresponding to the target text sample into the third mask model to obtain first negative sample feature information; determine the first semantic distance corresponding to the target text sample based on the sample feature information and the positive sample feature information; determine the second semantic distance corresponding to the target text sample based on the sample feature information and the first negative sample feature information; and perform self-supervised training on the text feature extraction model based on the first semantic distance and the second semantic distance corresponding to each text sample; wherein, the text feature extraction model is used to obtain the feature information of the input text based on the input text to match the retrieval text that is semantically similar to the input text, and different mask models correspond to different model parameters; the positive sample is the target text sample; or, the positive sample is generated by replacing key text information of the target text sample.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the training method of the self-supervised learning model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the training method of the self-supervised learning model as described in any one of claims 1 to 6.

Images