Information representation model processing method, information representation method and equipment

Abstract

The invention provides an information representation model processing method, an information representation method and equipment. An information bottleneck theory is applied to distillation of an information representation model; according to a correlation degree between a first feature representation of a sample output by an information representation model (student model) and a second feature representation of a sample output by a reference model (teacher model), and a correlation degree between the first feature representation after sample representation and a vector representation before representation, a training target of distillation training is constructed. The correlation degree between the feature representation output by the reference model and the information representation model is maximized, and the correlation degree between the feature representation output by the information representation model and the input data is minimized, so that the information representation model can learn important feature information, learning of unnecessary feature information is avoided, and the learning efficiency is improved. The risk of overfitting can be reduced, and the representation ability, performance and generalization ability of the information representation model after training are improved, so that the quality of information representation is improved.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a method for processing an information representation model, an information representation method and a device. Background Art

[0002] Information representation tasks refer to tasks that map information into digital vectors, including representation tasks of various types of information such as text representation, speech representation, image representation, and video representation, which can be applied to corresponding downstream tasks. For example, text representation can be applied to various downstream natural language processing tasks, such as text retrieval and clustering. Speech representation can be applied to downstream speech data processing tasks, such as speech recognition, classification, synthesis, and preprocessing. Image representation and video representation can be applied to downstream tasks in the field of computer vision, such as image recognition, classification, segmentation, object detection, and video editing.

[0003] In recent years, various pre-trained models have achieved great success in the field of information representation. However, the large-scale parameters and high-dimensional representation of various pre-trained models make the computational cost of model reasoning high and the reasoning speed slow, which brings major challenges to practical applications.

[0004] In order to reduce the inference cost of the pre-trained model and improve the inference speed, the model distillation method is usually used to train an information representation model (student model) with a smaller number of parameters using a pre-trained model with better performance (as a teacher model or reference model). However, traditional model distillation methods usually use maximum likelihood estimation or contrastive learning as the loss function, which has serious data noise and overfitting problems. Compared with the reference model, the obtained information representation model (student model) has serious performance and generalization problems. Summary of the invention

[0005] The present application provides an information representation model processing method, an information representation method and a device, so as to solve the problem that the information representation model obtained by model distillation has serious performance and generalization ability degradation.

[0006] In a first aspect, the present application provides a method for processing an information representation model, comprising: obtaining an information representation model to be trained, as well as a reference model and a training set for training; inputting samples in the training set into the information representation model and the reference model respectively, converting the samples into vector representations through the information representation model, characterizing the samples based on the vector representations of the samples to obtain a first feature representation of the samples, and characterizing the samples through the reference model to obtain a second feature representation of the samples; training the information representation model according to the vector representation, the first feature representation, and the second feature representation of the samples, with the training objectives of maximizing the correlation between the first feature representation and the second feature representation of the samples and minimizing the correlation between the first feature representation and the vector representation of the samples, to obtain a pre-trained information representation model.

[0007] In a second aspect, the present application provides an information representation method, comprising: receiving an information representation request sent by an end-side device, the information representation request containing input information to be represented, the input information including at least one type of information of text, voice, image, and video; inputting the input information into the information representation model for representation, and obtaining a feature representation of the input information, wherein the information representation model is trained in the following manner: obtaining a reference model and a training set for training, inputting samples in the training set into the information representation model and the reference model respectively, converting the samples into vector representations through the information representation model, representing the samples based on the vector representations of the samples, and obtaining a first feature representation of the samples, and representing the samples through the reference model, and obtaining a second feature representation of the samples; training the information representation model according to the vector representation, the first feature representation, and the second feature representation of the samples, with the goal of maximizing the correlation between the first feature representation and the second feature representation of the samples and minimizing the correlation between the first feature representation and the vector representation of the samples; and returning the feature representation of the input information to the end-side device.

[0008] In a third aspect, the present application provides a method for processing an information representation model, comprising: obtaining a pre-trained model for text representation, a text representation model to be trained, and a first text set; inputting the text in the first text set into the text representation model and the pre-trained model respectively, converting the text into a vector representation through the text representation model, performing text representation based on the vector representation of the text to obtain a first feature representation of the text, and representing the text through the pre-trained model to obtain a second feature representation of the text; according to the vector representation, the first feature representation, and the second feature representation of the text, with the training objectives of maximizing the correlation between the first feature representation and the second feature representation of the text and minimizing the correlation between the first feature representation and the vector representation of the text, training the text representation model to obtain a pre-trained text representation model.

[0009] In a fourth aspect, the present application provides an information representation method, comprising: receiving a text representation request sent by a terminal-side device, the text representation request comprising an input text to be represented; inputting the input text into the text representation model for representation to obtain a feature representation of the input text, wherein the text representation model is trained in the following manner: obtaining a pre-trained model for text representation, a text representation model to be trained, and a first text set; inputting the text in the first text set into the text representation model and the pre-trained model respectively, converting the text into a vector representation through the text representation model, performing text representation based on the vector representation of the text to obtain a first feature representation of the text, and representing the text through the pre-trained model to obtain a second feature representation of the text; training the text representation model according to the vector representation, the first feature representation, and the second feature representation of the text, with the training goal of maximizing the correlation between the first feature representation and the second feature representation of the text and minimizing the correlation between the first feature representation and the vector representation of the text; and returning the feature representation of the input text to the terminal-side device.

[0010] In a fifth aspect, the present application provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the server executes the method described in any of the preceding aspects.

[0011] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any of the preceding aspects is implemented.

[0012] The processing method, information representation method and device of the information representation model provided in the present application apply the information bottleneck theory to the distillation of the information representation model, and construct a training objective according to the correlation between the first feature representation output by the information representation model and the second feature representation output by the reference model, as well as the correlation between the first feature representation output by the information representation model and the vector representation before representation, so as to maximize the correlation between the feature representation output by the reference model and the information representation model, and minimize the correlation between the feature representation output by the information representation model and the input data, so that the information representation model can learn important feature information while avoiding learning unnecessary feature information, thereby reducing the risk of overfitting and improving the representation ability, performance and generalization ability of the information representation model after training. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] Figure 1 A schematic diagram of an exemplary system architecture applicable to the present application;

[0015] Figure 2 A flow chart of a model distillation method provided for an exemplary embodiment of the present application;

[0016] Figure 3 A flow chart of a method for calculating a first loss function provided by an exemplary embodiment of the present application;

[0017] Figure 4 A framework diagram of model distillation training provided for an exemplary embodiment of the present application;

[0018] Figure 5 A flow chart of a method for fine-tuning training of a student model provided by an exemplary embodiment of the present application;

[0019] Figure 6 A flow chart of a method for calculating a second loss function provided by an exemplary embodiment of the present application;

[0020] Figure 7 A framework diagram of fine-tuning training of a student model provided by an exemplary embodiment of the present application;

[0021] Figure 8 An exemplary flow chart of a two-stage model distillation method provided for an exemplary embodiment of the present application;

[0022] Fig. 9 A flowchart of an information representation method provided for an exemplary embodiment of the present application;

[0023] Fig.10A flow chart of a text representation model distillation method provided for an exemplary embodiment of the present application;

[0024] Fig.11 A flowchart of a method for fine-tuning a text representation model provided for an exemplary embodiment of the present application;

[0025] Fig.12 A flowchart of a text representation method provided for an exemplary embodiment of the present application;

[0026] Fig.13 A schematic diagram of the structure of a server provided in an embodiment of the present application.

[0027] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0029] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] First, the terms involved in this application are explained:

[0031] Information Bottleneck (IB): A representation learning theory that holds that representation learning needs to strike a balance between retaining correct output information and filtering irrelevant input information. In this application, the information bottleneck theory is applied to the field of information representation learning, and the information bottleneck theory is used to explain the dilemma of information representation learning: the feature representation Z of information obtained based on model representation can accurately predict the label Y of the information, which requires that the feature representation Z contains sufficient feature information, but should not retain too much irrelevant feature information of the initial vector representation X of the information. Based on the information bottleneck theory, this balance is achieved by maximizing the mutual information I(Z,Y) between the feature representation Z and the label Y, and minimizing the mutual information I(Z,X) between the feature representation Z and the vector representation X.

[0032] Mutual Information (MI): The amount of information shared by two random variables, reflecting the statistical dependence between the two variables, can be obtained by calculating the KL divergence (Kullback-Leibler Divergence) of the joint distribution and marginal distribution of the two variables. I(Z,X) is usually used to represent the mutual information between two random variables Z and X. When I(Z,X) = 0, it means that there is no dependence between the two random variables and the two random variables are relatively independent.

[0033] Hilbert-Schmidt Independence Criterion (HSIC): A statistical method that evaluates the independence of two random variables by calculating the Hilbert-Schmidt norm. HSIC can effectively evaluate the independence of two random variables. Its property is that the lower the HSIC value, the more independent the two variables are and the lower the mutual information between the two variables. When I(Z,X)=0, HSIC(Z,X)=0.

[0034] Information Retrieval (IR) is a technology for quickly finding relevant information from large-scale unstructured information collections.

[0035] Text Representation (TR): A technique for mapping text into digital vectors, which can be used for downstream Natural Language Processing (NLP) tasks such as text retrieval, clustering, and classification.

[0036] Pre-trained Model (PTM): A model pre-trained on a large-scale corpus that can be migrated to downstream tasks.

[0037] Semantic Similarity: Measures the semantic closeness between two texts.

[0038] Multimodal tasks: refers to downstream tasks whose input and output data involve multiple modal data such as images and text, such as visual question answering tasks, image description tasks, visual implication tasks, referential expression and understanding tasks, image generation tasks, etc.

[0039] Multi-modal Pre-training Model: refers to a pre-training model whose input and output data involve multiple modal data such as images and texts. After fine-tuning and training, it can be applied to multi-modal task processing.

[0040] Pre-trained language model: a pre-trained model obtained after pre-training the language model (Language Model, LM for short).

[0041] A large model refers to a deep student model with large-scale model parameters, usually containing hundreds of millions, tens of billions, or even hundreds of billions of model parameters. A large model can also be called a foundation model / foundation model (FM). It is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLMs) and multimodal pre-trained models.

[0042] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing, computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering, image description, image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0043] In recent years, various pre-trained models have achieved great success in the field of information representation. However, the large-scale parameters and high-dimensional representation of various pre-trained models make the computational cost of model reasoning high and the reasoning speed slow, which brings major challenges to practical applications.

[0044] In order to reduce the inference cost of the pre-trained model and improve the inference speed, the model distillation method is usually used to train an information representation model (student model) with a small number of parameters using the pre-trained model (as a teacher model or reference model). However, traditional model distillation methods usually use maximum likelihood estimation or contrastive learning as the loss function, which has serious data noise and overfitting problems. Compared with the reference model, the obtained information representation model (student model) has serious performance and generalization problems.

[0045] In response to the technical problem that the information representation model obtained by training with the traditional model distillation method has serious performance and generalization ability degradation, the present application provides a method for processing an information representation model. For the information representation model to be trained, based on the reference model and the training set used for training, the samples in the training set are respectively input into the information representation model to be trained and the reference model, the samples are converted into vector representations by the information representation model, the samples are characterized based on the vector representations of the samples to obtain the first feature representation of the samples, and the samples are characterized by the reference model to obtain the second feature representation of the samples; according to the vector representation, the first feature representation and the second feature representation of the samples, the information representation model is trained to maximize the correlation between the first feature representation and the second feature representation of the samples and minimize the correlation between the first feature representation and the vector representation of the samples as the training objectives, and the pre-trained information representation model is obtained. Among them, the correlation between the first feature representation and the second feature representation of the samples can be measured using the mutual information between the first feature representation and the second feature representation of the samples, and of course other information can also be used to measure the correlation between the first feature representation and the second feature representation of the samples. The correlation between the first feature representation and the vector representation of a sample can be measured using the mutual information between the first feature representation and the vector representation of the sample. Of course, other information can also be used to measure the correlation between the first feature representation and the vector representation of the sample.

[0046] The reference model may be various pre-trained models for information representation, which have larger parameters than the information representation model to be trained. For example, a pre-trained model for text representation, a pre-trained model for image representation, a pre-trained model for voice data representation, and a pre-trained model for representing other types of data (such as video) or multimodal data, which may be a pre-trained language model, a large-scale pre-trained model, a multimodal pre-trained model, etc., which is not specifically limited in this embodiment.

[0047] The information representation model to be trained is a lightweight model with the same task capability as the reference model, and has fewer parameters than the reference model. The information representation model to be trained can be obtained by compressing the reference model using model compression methods such as pruning, quantization, low-rank decomposition, and parameter sharing, or it can be a reconstructed model, which is not specifically limited in this embodiment. For example, when the information representation model to be obtained is a lightweight text representation model, various pre-trained language models can be used as reference models.

[0048] The processing method of the information representation model of this embodiment can realize the distillation training of the information representation model. In the distillation training, the information representation model to be trained is used as the student model, and the reference model is used as the teacher model. In this embodiment, the information bottleneck theory is applied to the distillation training of the information representation model. According to the mutual information between the first feature representation output by the information representation model (student model) and the second feature representation output by the reference model (teacher model), and the mutual information between the first feature representation output by the information representation model (student model) and the vector representation before representation, a first loss function used in the distillation training is constructed. By minimizing the first loss function, the mutual information between the feature representation output by the reference model (teacher model) and the information representation model (student model) is maximized, and the mutual information between the feature representation output by the information representation model (student model) and the input data is minimized, so that the information representation model can learn important feature information and avoid learning unnecessary feature information, which can reduce the risk of overfitting, thereby improving the representation ability, performance and generalization ability of the information representation model after training.

[0049] The method of this embodiment can be used to perform model distillation on representation models of various types of data to obtain lightweight representation models while ensuring the performance and generalization ability of the representation models. When training representation models of different types of data, the samples in the training set used are data samples of the corresponding type. For example, when training a text representation model, the samples in the training set are text; when training a speech representation model, the samples in the training set are speech; when training an image representation model, the samples in the training set are images; when training a multimodal information representation model, the samples in the training set include samples of multiple modalities.

[0050] The information representation model obtained by the method of this embodiment can be a lightweight representation model for various types of data, which is used in downstream tasks to characterize the corresponding types of input data and obtain the feature representation of the input data. For example, the information representation model can be a text representation model, which is used to characterize the input text and obtain the feature representation of the input text; the information representation model can be a speech representation model, which is used to characterize the input speech and obtain the feature representation of the input speech; the information representation model can be an image representation model, which is used to characterize the input image and obtain the feature representation of the input image; the information representation model can be a multimodal information representation model, which can be used to characterize information of multiple preset different types (modalities).

[0051] Figure 1 FIG. 1 is a schematic diagram of an exemplary system architecture applicable to the present application. Figure 1 As shown, the system architecture includes a server and a terminal device. There is a communication link between the server and the terminal device, which can realize the communication connection between the server and the terminal device.

[0052] The server is a device with computing capabilities deployed in the cloud or locally, such as a cloud cluster. The server is responsible for acquiring and storing the information representation model and reference model to be trained, as well as the training set for training in the model distillation phase, and training the information representation model based on the training set and the reference model to obtain a pre-trained information representation model. The information representation model can be a representation model of various types of information, including but not limited to a text representation model, a speech representation model, an image representation model, and a multimodal representation model.

[0053] The end-side device can be an electronic device that runs downstream applications, and specifically can be a hardware device with network communication functions, computing functions, and information display functions, including but not limited to smartphones, tablet computers, desktop computers, local servers, cloud servers, etc. The end-side device needs to use the representation capabilities of the information representation model when running downstream applications. The downstream applications run by the end-side device can be natural language processing tasks such as document retrieval, recommendation, and classification, or speech processing tasks such as speech recognition, classification, and synthesis, or image processing tasks such as image recognition, segmentation, and detection, or multimodal information processing tasks based on multimodal information such as text, speech, and images. When implementing the functions of downstream applications, it is necessary to use the representation capabilities of the information representation model to represent a given type of input information into a feature representation, and implement the processing tasks of the downstream application based on the feature representation of the information.

[0054] based on Figure 1In the system architecture shown, the end-side device sends the input information to be represented to the server. The server receives the input information to be represented, characterizes the input information into the information representation model, obtains the feature representation of the input information, and returns the feature representation of the input information to the end-side device. The end-side device receives the feature representation of the input information returned by the server, and based on the feature representation of the input information, continues to execute subsequent processing logic to implement the processing tasks of the downstream application.

[0055] In addition, the server can also provide the trained information representation model (or the parameters of the trained information representation model) to the end-side device, and the end-side device can use the trained information representation model locally to implement specific downstream application tasks based on the information representation capability of the information representation model, such as natural language processing tasks such as document retrieval, recommendation, and classification, or speech processing tasks such as speech recognition, classification, and synthesis, or image processing tasks such as image recognition, segmentation, and detection, or multimodal information processing tasks based on multimodal information such as text, speech, and images. Of course, the trained information representation model can also be applied to other tasks such as those in the field of artificial intelligence, which are not specifically limited here.

[0056] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0057] Figure 2 This is a flow chart of a model distillation method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. Figure 2 As shown, the specific steps of this method are as follows:

[0058] Step S200: Obtain an information representation model to be trained, as well as a reference model and a training set for training.

[0059] In this embodiment, the reference model can be various pre-trained models for information representation, which have larger parameters than the information representation model to be trained. For example, a pre-trained model for text representation, a pre-trained model for image representation, a pre-trained model for voice data representation, and a pre-trained model for representing other types of data (such as video) or multimodal data, specifically a BERT (Bidirectional Encoder Representations from Transformers, based on Transformer bidirectional encoder representation) model, a pre-trained language model, a large-scale pre-trained model, a multimodal pre-trained model, etc., which is not specifically limited in this embodiment.

[0060] The information representation model to be trained is a lightweight model with the same task capability as the reference model, and has fewer parameters than the reference model. The information representation model to be trained can be obtained by compressing the reference model using a model compression method, or it can be a reconstructed model, for example, it can be obtained by pruning, quantization, low-rank decomposition, parameter sharing or other model compression methods, which are not specifically limited in this embodiment. For example, when the information representation model to be obtained is a lightweight text representation model, various pre-trained language models can be used as reference models, and the reference model can be compressed to obtain the information representation model.

[0061] The training set used for model distillation training includes large-scale unsupervised data. According to the specific task capabilities of the information representation model, when it is applied to the representation of specific types of information (such as text, speech, images, multimodal information, etc.), the corresponding type of training set is used for distillation training. For example, assuming that the information representation model is a text representation model, the training set contains large-scale text corpus; assuming that the information representation model is an image representation model, the training set contains large-scale image samples. The samples in the training set can come from public data sets, can be collected by the server, or can be provided by the user through the terminal device, which is not specifically limited in this embodiment.

[0062] Exemplarily, in an application scenario, a user specifies a pre-trained model for information representation to a server through a terminal device, and requests the server to train and obtain a lightweight information representation model with the same information representation capability. The server obtains the pre-trained model specified by the terminal device as a reference model, compresses the pre-trained model, and obtains the information representation model. The server obtains the corresponding training set according to the information type targeted by the representation capability of the information representation model, and performs distillation training on the information representation model based on the training set and the reference model, thereby obtaining an information representation model with better performance.

[0063] Furthermore, the server may provide the information representation model to the terminal device, so that the terminal device can store the information representation model locally and run it locally. Alternatively, the server may provide the terminal device with an application programming interface (Application Program Interface, referred to as API) of the information representation model, so that the terminal device can use the information representation capability of the information representation model by calling the API of the information representation model.

[0064] Optionally, the user specifies a pre-trained model as a reference model through the terminal device, which can be achieved in the following way: the server sends the optional pre-trained models corresponding to each type of information and the relevant introduction information of each pre-trained model to the terminal device. Among them, the optional pre-trained models include but are not limited to: pre-trained text representation model, image representation model, speech representation model, multimodal representation model. The terminal device displays the optional pre-trained models corresponding to each type of information provided by the server and the relevant introduction information of each pre-trained model through the front-end interface, so that the user can browse and specify one of the pre-trained models.

[0065] Optionally, the user may send a specified pre-trained model for information representation to the server through the client device, or send a specified pre-trained model for information representation and a training set for distillation training to the server. The server receives the specified pre-trained model and training set.

[0066] During the distillation training of the information representation model, the server uses the training set and the reference model to iteratively perform multiple rounds of distillation training on the information representation model, and obtains the pre-trained information representation model after the iteration. In one iterative training, the server samples samples of a preset batch size (represented by l, where l is a positive integer) from the training set, and uses the sampled batch of samples for training.

[0067] Step S201: Input the samples in the training set into the information representation model and the reference model respectively, convert the samples into vector representations through the information representation model, characterize the samples based on the vector representations of the samples to obtain the first feature representation of the samples, and characterize the samples through the reference model to obtain the second feature representation of the samples.

[0068] During an iterative training process, the server inputs the samples used for this round of iteration into the information representation model and the reference model respectively, converts the samples into vector representations through the information representation model, continues to characterize the vector representations of the samples, obtains and outputs the feature representations of the samples; converts the samples into vector representations through the reference model, and continues to characterize the vector representations of the samples, obtains and outputs the feature representations of the samples. In this embodiment, in order to distinguish the feature representations output by the two models, the first feature representation refers to the feature representation output by the information representation model, and the second feature representation refers to the feature representation output by the reference model. In this way, the vector representation of the samples converted by the information representation model, the first feature representation of the samples characterized by the information representation model, and the second feature representation of the samples characterized by the reference model can be obtained.

[0069] It should be noted that both the information representation model and the reference model have the function of first converting the input information into a vector representation (i.e., embedding), and then characterizing the vector representation. Among them, the conversion of the input information into a vector representation is implemented by the first layer of the information representation model and the reference model. The first layer usually does not contain learnable / trainable parameters and does not change during the distillation training process. The vector representation of the sample output by the first layer of the information representation model. In this embodiment, when training the information representation model, the training target is constructed based on the vector representation of the sample converted by the information representation model to be trained, and the feature representations respectively characterized by the two models, and the information representation model is distilled and trained, and the vector representation of the sample converted by the reference model is not used.

[0070] Step S202: According to the vector representation, the first feature representation and the second feature representation of the sample, the information representation model is trained to obtain a pre-trained information representation model by maximizing the correlation between the first feature representation and the second feature representation of the sample and minimizing the correlation between the first feature representation and the vector representation of the sample as the training objectives.

[0071] The correlation between the first feature representation and the second feature representation of the sample can be measured using the mutual information between the first feature representation and the second feature representation of the sample. Of course, other information can also be used to measure the correlation between the first feature representation and the second feature representation of the sample. The correlation between the first feature representation and the vector representation of the sample can be measured using the mutual information between the first feature representation and the vector representation of the sample. Of course, other information can also be used to measure the correlation between the first feature representation and the vector representation of the sample.

[0072] In this embodiment, the information bottleneck theory is applied to the distillation training of the information representation model. The first loss function used in the distillation training is constructed according to the mutual information between the first feature representation output by the information representation model and the second feature representation output by the reference model, as well as the mutual information between the vector representation of the sample before the information representation model is represented and the first feature representation of the sample after the representation. By minimizing the first loss function, the mutual information between the feature representation represented by the reference model and the information representation model is maximized, while the mutual information between the feature representation represented by the information representation model and the input data is minimized. This allows the information representation model to learn important feature information in the input information while avoiding learning unnecessary feature information. This can reduce the risk of overfitting, thereby improving the representation ability, performance and generalization ability of the trained information representation model.

[0073] The first loss function consists of two parts: one is the estimated value of the mutual information between the first feature representation and the second feature representation of the sample, expressed as L ev1The other part is the estimated value of the mutual information between the first feature representation and the vector representation of the sample, expressed as L ev2 Based on the estimated values ​​of the two mutual information, a first loss function is constructed. Exemplarily, the first loss function can be: L 1 =-(L ev1 -βL ev2 ), where β is a hyperparameter, which can be configured and adjusted according to the needs of the actual application scenario. The value of β is a positive number, for example, β can be 1.0, 0.8, 1.5, etc., which is not specifically limited here.

[0074] In addition, the first loss function can also be: L 1 =L ev2 -αL ev1 , α is a configurable hyperparameter. The first loss function can also be constructed in other forms by minimizing the first loss function L 1 , which can maximize the estimated value L of the mutual information between the first feature representation and the second feature representation of the sample ev1 , and minimize the estimated value L of the mutual information between the first feature representation and the vector representation of the sample ev2 The effect is sufficient, and this embodiment is not specifically limited here.

[0075] Among them, the estimated value L of the mutual information between the first feature representation and the second feature representation of the sample ev1 , which can be calculated by solving the KL divergence of the joint distribution and marginal distribution of the first feature representation and the second feature representation of the sample. Optionally, the Mutual Information Neural Estimation (MINE) method or other mutual information lower bound estimation methods can be used to estimate the lower bound of the mutual information between the first feature representation and the second feature representation of the sample as the estimated value L ev1 .

[0076] Optionally, in order to reduce the amount of calculation of the mutual information between the first feature representation and the second feature representation of the sample, the contrastive learning loss can be used as an estimate of the lower bound of the mutual information between the first feature representation and the second feature representation of the sample, as the estimated value L ev1 , which can be implemented by using a commonly used contrastive learning loss function, and is not specifically limited here.

[0077] The estimated value L of the mutual information between the first feature representation and the vector representation of the sample ev2, which can be calculated by solving the KL divergence of the joint distribution and marginal distribution of the first feature representation and the vector representation of the sample. Optionally, the upper bound of the mutual information between the first feature representation and the vector representation of the sample can be estimated by using the contrastive log-ratio upper bound (CLUB) or other mutual information upper bound estimation methods to estimate the upper bound of the mutual information between the first feature representation and the vector representation of the sample as the estimated value L ev2 .

[0078] Optionally, in order to reduce the amount of calculation of the mutual information between the first feature representation and the vector representation of the sample, the mutual information between the first feature representation and the vector representation of the sample can be estimated by calculating the correlation information between the first feature representation and the vector representation of the sample. Exemplarily, a HSIC-based method can be used to calculate the HSIC value between the first feature representation and the vector representation of the sample as the upper bound estimate L of the mutual information between the first feature representation and the vector representation of the sample. ev2 .

[0079] In this embodiment, the information bottleneck theory is applied to the distillation of the information representation model. The first loss function used in distillation training is constructed according to the mutual information between the first feature representation output by the information representation model and the second feature representation output by the reference model, as well as the mutual information between the first feature representation output by the information representation model and the vector representation before representation. By minimizing the first loss function, the mutual information between the feature representation output by the reference model and the information representation model is maximized, while the mutual information between the feature representation output by the information representation model and the input data is minimized. This allows the information representation model to learn important feature information while avoiding learning unnecessary feature information. This can reduce the risk of overfitting, thereby improving the representation ability, performance and generalization ability of the information representation model after training, thereby improving the quality of information representation.

[0080] In an optional embodiment, the first loss function used in the training process of the model distillation includes two parts: one part is a contrastive learning loss based on the first feature representation and the second feature representation of the sample (referred to as the first contrastive learning loss), which is an estimate of the lower bound of the mutual information between the first feature representation and the second feature representation of the sample; the other part is the correlation information between the first feature representation and the vector representation of the sample (referred to as the first correlation information), which is an estimate of the upper bound of the mutual information between the first feature representation and the vector representation of the sample.

[0081] Specifically, Figure 3 As shown, the aforementioned step S202 can be implemented in the following manner:

[0082] Step S2021: Calculate a first contrastive learning loss based on the first feature representation and the second feature representation of the sample as an estimated value of the mutual information between the first feature representation and the second feature representation of the sample.

[0083] In this embodiment, for any sample i, the first feature representation and the second feature representation of sample i are combined into a positive sample pair, and the first feature representation of sample i and the second feature representation of any sample j are combined into a negative sample pair, where sample j and sample i are different samples; the similarity of the positive sample pair and the similarity of the negative sample pair are calculated; based on the similarity of the positive sample pair and the similarity of the negative sample pair, the first contrastive learning loss is calculated.

[0084] For example, let S represent the set of first feature representations of the sample, and let s i represents the first characteristic representation of sample i (i.e., the i-th sample). T represents the set of second characteristic representations of the sample, and t i Represents the second feature representation of sample i (i.e., the i-th sample), t j represents the second feature representation of sample j (i.e., the jth sample). Let X represent the set of vector representations of the previous sample, and x i represents the vector representation of the previous sample i (i.e. the i-th sample), x j Represents the vector representation of the previous sample j (i.e. the jth sample).

[0085] In this step, the following formula (1) can be used to calculate the first contrastive learning loss:

[0086]

[0087] Among them, L cl represents the first contrastive learning loss. sim(s i ,t i ) means s i With t i The similarity between i ,t j ) means s i With t j exp represents the exponential function with the natural constant e as the base. l is the number of samples participating in the training, that is, the batch size sampled during the iterative training process.

[0088] Optionally, the similarity between the first feature representation and the second feature representation of the sample may be the cosine similarity of the two feature representations. in, Indicates S iIn addition, the similarity between the first feature representation and the second feature representation of the sample can also be calculated using other vector similarity calculation methods, such as Euclidean distance, Manhattan distance, etc., which are not specifically limited in this embodiment.

[0089] The first contrastive learning loss function of formula (1) can be used as an estimate of the lower bound of the mutual information between the first feature representation and the second feature representation of the sample. By maximizing the second contrastive learning loss, the mutual information between the first feature representation and the second feature representation of the sample can be maximized.

[0090] In another optional embodiment, for any sample i, the first feature representation and the second feature representation of sample i are used to form a positive sample pair, and a preset number of other samples j are randomly selected as negative samples of sample i to form a negative sample pair with sample i; the similarity of the positive sample pair and the similarity of the negative sample pair are calculated; based on the similarity of the positive sample pair and the similarity of the negative sample pair, the first contrastive learning loss is calculated.

[0091] Step S2022: Calculate first correlation information between the vector representation and the first feature representation of the sample as an estimated value of the mutual information between the first feature representation and the vector representation of the sample.

[0092] In this embodiment, the upper bound of the mutual information between the first feature representation and the vector representation of the sample is estimated by calculating the correlation information between the vector representation and the first feature representation of the sample, which is used as the estimated value of the mutual information between the first feature representation and the vector representation of the sample.

[0093] In an optional implementation of this step, based on the Hilbert-Schmidt independence criterion HSIC, the HSIC value of the vector representation and the first feature representation of the sample is calculated as an estimated value of the mutual information between the first feature representation and the vector representation of the sample.

[0094] Exemplarily, the following formula (2) may be used to calculate the HSIC value of the vector representation X and the first feature representation S of the sample:

[0095]

[0096] Among them, HSIC(X,S) represents the HSIC value of the sample vector representation X and the first feature representation S. tr() represents the rank of the matrix. H = I-1 / l, I is the unit matrix, l is the number of samples involved in the training. K X and K S are the Gram matrices of X and S respectively. X Each of k() is the Radial Basis Function (RBF) kernel, k(xi ,x j ) = exp(γ‖x i ,x j ‖). K S Each of k(S i ,S j ) = exp(γ‖S i ,S j ‖). γ is a hyperparameter with a value range of (0,1], such as γ=0.2. ‖ ‖ represents the second norm. Among them, K X and K S Any existing method for solving the Gram matrix (gram matrix) may be used for implementation, and this embodiment is not specifically limited here.

[0097] In another optional implementation of this step, other methods of calculating the correlation between two sets of vector representations, such as calculating the correlation coefficient, can be used to calculate the first correlation information between the vector representation of the sample and the first feature representation. This is not limited to the HSIC-based method.

[0098] Step S2023: Calculate a first loss function according to the first contrastive learning loss and the first correlation information, where the first loss function is used to increase the first contrastive learning loss and reduce the first correlation information.

[0099] In this step, a first loss function is constructed according to the first contrastive learning loss and the first correlation information, so as to maximize the first contrastive learning loss by minimizing the first loss function and minimize the first correlation information. The first loss function can be used as follows: 1 =-(L cl -β 1 HSIC(X,S)). Where β 1 is a hyperparameter, which can be configured and adjusted according to the needs of the actual application scenario. 1 The value is a positive number, such as β 1 It can be 1.0, etc., and is not specifically limited here.

[0100] Step S2024: According to the first loss function, the training information represents the parameters of the model.

[0101] After calculating the first loss function, the parameters of the information representation model are updated through back propagation according to the first loss function. After multiple rounds of iterative training, the trained information representation model can be obtained. Since a large-scale unsupervised training set is used to perform distillation training on the information representation model, the trained information representation model is used as a pre-trained information representation model, and a supervised training set can be used for fine-tuning training later.

[0102] The method of this embodiment estimates the lower bound of the mutual information between the first feature representation and the second feature representation of the sample by calculating the first contrastive learning loss based on the first feature representation and the second feature representation of the sample; estimates the upper bound of the mutual information between the first feature representation and the vector representation of the sample by calculating the HSIC value between the vector representation of the sample and the first feature representation, and constructs a first loss function including the first contrastive learning loss and the HSIC value. By minimizing the first loss function, the training goal of maximizing the mutual information between the feature representation output by the reference model and the information representation model and minimizing the mutual information between the feature representation output by the information representation model and the input data can be achieved, so that the information representation model can learn important feature information while avoiding learning unnecessary feature information, which can reduce the risk of overfitting, thereby improving the representation ability, performance and generalization ability of the information representation model after training, thereby improving the quality of information representation.

[0103] For example, Figure 4 The framework diagram of the model distillation training provided in this embodiment is as follows: Figure 4 As shown, the vector representation X of the sample before characterization is represented by the information representation model and the reference model respectively, and the first feature representation S after the information representation model characterization and the second feature representation T after the reference model characterization are obtained. Based on the first feature representation S after the information representation model characterization and the second feature representation T after the reference model characterization, the first contrastive learning loss L is calculated. cl , as the estimated value of the upper bound of the mutual information between the two. Based on the first feature representation S after characterization by the information representation model and the vector representation X of the sample before characterization, the HSIC value L of the two is calculated hsic , as the estimated value L of the lower bound of the mutual information between the two ev1 . Furthermore, the first loss function of distillation training can be: L 1 =-(L cl -β 1 L hsic ), β 1 is a hyperparameter and can be set to 1.0. By minimizing the first loss function L 1 , to maximize the first contrastive learning loss L cl , and minimize the HSIC value L hsic , thereby maximizing the mutual information between the first feature representation S after characterization by the information representation model and the second feature representation T after characterization by the reference model, and minimizing the mutual information between the first feature representation S after characterization by the information representation model and the vector representation X of the sample before characterization.

[0104] In an optional embodiment, after obtaining the pre-trained information representation model, the server may provide the information representation model to the end-side device. Exemplarily, in the case where the end-side device stores the structure of the information representation model, the server may send the trained parameters of the information representation model to the end-side device. The end-side device initializes the parameters of the locally stored information representation model according to the trained parameters of the information representation model to obtain the pre-trained information representation model. Exemplarily, the server may also send download information of the information representation model to the end-side device. The end-side device downloads the pre-trained information representation model to the local device according to the received download information of the information representation model.

[0105] In this embodiment, the server provides the pre-trained information representation model obtained through distillation training to the terminal side device, so that the terminal side device can store the pre-trained information representation model locally. The terminal side device can use the pre-trained information representation model locally to implement the information representation function, thereby improving the quality of information representation.

[0106] In an optional embodiment, after obtaining the pre-trained information representation model, the server may provide information representation services to the terminal device based on the pre-trained information representation model.

[0107] Exemplarily, when the end-side device needs to represent information, it sends an information representation request to the server, and the information representation request includes the input information to be represented. The server receives the information representation request sent by the end-side device, obtains the input information to be represented, inputs the input information into the information representation model to characterize, and obtains the feature representation of the input information. The server returns the feature representation of the input information to the end-side device. The end-side device receives the feature representation of the input information returned by the server, and executes subsequent processing logic based on the feature representation of the input information.

[0108] Exemplarily, the server may also provide the end-side device with an API of a pre-trained information representation model. When the end-side device needs to represent information, it sends an API call request to the server, and the API call request contains the input information to be represented. When the server responds to the API call request, it obtains the input information to be represented, inputs the input information into the information representation model for characterization, and obtains a feature representation of the input information. The server returns the feature representation of the input information to the end-side device. The end-side device receives the feature representation of the input information returned by the server, and executes subsequent processing logic based on the feature representation of the input information.

[0109] In this embodiment, the server can provide information representation capabilities to the external terminal device based on the pre-trained information representation model obtained through distillation training, which can improve the quality of information representation.

[0110] Based on any of the foregoing method embodiments, in an optional embodiment, after obtaining the pre-trained information representation model through model distillation, the server can also use a high-quality labeled data set to fine-tune the pre-trained information representation model to further improve the representation ability and performance of the information representation model. Figure 5 A flow chart of the method for fine-tuning the information representation model provided in this embodiment, such as Figure 5 As shown, the specific steps of this method are as follows:

[0111] Step S501: Obtain a labeled data set, where the training data in the labeled data set includes: training samples, positive samples and negative samples of pre-labeled training samples.

[0112] In this embodiment, the labeled data set used for fine-tuning the pre-trained information representation model is a supervised data set. The labeled data set includes not only training samples, but also positive and negative samples of pre-labeled training samples. The labeled data set can use a public data set; it can also be obtained by collecting large-scale unlabeled samples by the server and labeling positive and negative samples; it can also be provided by the user through the terminal device, which is not specifically limited in this embodiment.

[0113] Different from the training set used in the distillation training stage, the fine-tuning training stage uses high-quality supervised data with labeled positive and negative samples for training, which can further improve the representation ability of the information representation model and thus improve the quality of information representation.

[0114] Each training data in the labeled data set consists of three parts: a training sample, a positive sample of the pre-labeled training sample, and multiple negative samples of the pre-labeled training sample.

[0115] In an optional application scenario, the annotated dataset may be an annotated dataset of a vertical field provided by the end-side device. The server uses the annotated dataset of the vertical field to fine-tune the pre-trained information representation model to obtain an exclusive information representation model suitable for the vertical field. The server provides the trained exclusive information representation model of the vertical field to the end-side device.

[0116] In the fine-tuning training phase, the server iterates the information representation model for multiple rounds of fine-tuning training, and obtains the trained information representation model after the iteration. In one iterative training, the server samples training data of a preset batch size (represented by l, where l is a positive integer) from the labeled data set, and uses the sampled batch of training data for training.

[0117] Step S502: input the training samples and the positive samples and negative samples into the pre-trained information representation model for conversion and representation, and obtain the vector representation and feature representation of the training samples, as well as the feature representation of the positive samples and negative samples of the training samples.

[0118] During an iterative training process, the server inputs the training samples and positive and negative samples in the training data used for this round of iterative training into the pre-trained information representation model, converts the training samples into vector representations and represents them through the pre-trained information representation model, and outputs the vector representation and feature representation of the training samples; converts and represents the positive and negative samples respectively through the pre-trained information representation model, and outputs the feature representation of the positive and negative samples.

[0119] Step S503, calculate a second loss function based on the vector representation and feature representation of the training sample, and the feature representation of the positive sample and the negative sample of the training sample, wherein the second loss function is used to increase the first mutual information between the feature representation of the training sample and the feature representation of the positive sample, reduce the second mutual information between the feature representation of the training sample and the feature representation of the negative sample, and reduce the third mutual information between the feature representation of the training sample and the vector representation.

[0120] In this embodiment, the information bottleneck theory is applied to the fine-tuning training of the information representation model. The second loss function used in the fine-tuning training is constructed according to the mutual information between the feature representation of the training sample after characterization and the feature representation of the comparison sample (including positive and negative samples), and the mutual information between the feature representation of the training sample after characterization and the vector representation of the training sample before characterization. By minimizing the second loss function, the mutual information between the feature representation of the training sample and the feature representation of the positive sample is increased (denoted as the first mutual information), the mutual information between the feature representation of the training sample and the feature representation of the negative sample is reduced (denoted as the second mutual information), and the mutual information between the feature representation of the training sample and the vector representation is reduced (denoted as the third mutual information). This enables the information representation model to learn important feature information in the input information while avoiding learning unnecessary feature information. This can reduce the risk of overfitting, improve the representation ability, performance and generalization ability of the information representation model after training, and thus improve the quality of information representation.

[0121] The second loss function consists of two parts: one is the second contrastive learning loss of the feature representation of the training sample, expressed as L′ cl The other part is the estimated value of the mutual information between the feature representation of the training sample after representation and the vector representation before representation, represented by L′ ev2 Indicates that this part is related to L in the distillation training stage ev2 The calculation method is the same. Based on the second contrastive learning loss and L′ ev2 , construct the second loss function. Exemplarily, the second loss function can be: L 2 =-(L′ cl -β 2 L′ ev2 ), where β 2is a hyperparameter, which can be configured and adjusted according to the needs of the actual application scenario. 2 The value is a positive number, such as β 2 It can be 1.0, 0.8, 1.5, etc., and is not specifically limited here.

[0122] In addition, the second loss function can also be: L 2 =L′ ev2 -α 2 L′ cl , α 2 is a configurable hyperparameter. The second loss function can also be constructed in other forms by minimizing the second loss function L 2 , which can maximize the second contrastive learning loss and minimize the estimated value L′ of the mutual information between the feature representation of the training sample after characterization and the vector representation before characterization ev2 The effect is sufficient, and this embodiment is not specifically limited here.

[0123] Among them, the estimated value L′ of the mutual information between the feature representation after training sample characterization and the vector representation before characterization ev2 , and the estimated value L of the mutual information between the first feature representation of the sample after being represented by the information representation model and the vector representation of the sample before being represented in the aforementioned step S202 ev2 , is calculated in the same way, please refer to the relevant content in the aforementioned method embodiment for details, and this embodiment will not be repeated here.

[0124] Step S504: train the parameters of the information representation model according to the second loss function to obtain a trained information representation model.

[0125] After calculating the second loss function, the parameters of the information representation model are updated through back propagation according to the second loss function. After multiple rounds of iterative training, the trained information representation model can be obtained. Since a high-quality supervised data set is used to fine-tune the information representation model, the information representation ability of the information representation model can be further improved, thereby improving the quality of information representation.

[0126] In this embodiment, the information bottleneck theory is applied to the fine-tuning training of the information representation model. The second loss function used for fine-tuning training is constructed according to the mutual information between the feature representation of the training sample after characterization and the feature representation of the comparison sample (including positive and negative samples), and the mutual information between the feature representation of the training sample after characterization and the vector representation of the training sample before characterization. By minimizing the second loss function, the first mutual information between the feature representation of the training sample and the feature representation of the positive sample is increased, the second mutual information between the feature representation of the training sample and the feature representation of the negative sample is reduced, and the third mutual information between the feature representation of the training sample and the vector representation is reduced, so that the information representation model can learn important feature information in the input information while avoiding learning unnecessary feature information, which can reduce the risk of overfitting, improve the representation ability, performance and generalization ability of the information representation model after training, and thus improve the quality of information representation.

[0127] In an optional embodiment, similar to the first loss function used in the model distillation stage, the second loss function used in the fine-tuning training stage of the information representation model also includes two parts: one part is a second contrastive learning loss calculated based on the feature representation of the training sample and the feature representation of the positive and negative samples of the training sample, and the second contrastive learning loss is positively correlated with the first mutual information and negatively correlated with the second mutual information. The other part is the correlation information between the feature representation and the vector representation of the training sample (referred to as the second correlation information), which is used as an estimate of the upper bound of the mutual information between the feature representation and the vector representation of the training sample.

[0128] Specifically, Figure 6 As shown, the aforementioned step S503 can be implemented in the following manner:

[0129] Step S5031: Calculate a second contrastive learning loss based on the feature representation of the training sample and the feature representation of the positive sample and the negative sample of the training sample, wherein the second contrastive learning loss is positively correlated with the first mutual information and negatively correlated with the second mutual information.

[0130] In this embodiment, since the positive samples and negative samples corresponding to each training sample are marked in the labeled data set, in an optional implementation method of this step, the training sample and its positive sample can be combined into a positive sample pair, and the training sample and its negative sample can be combined into a negative sample pair. In addition, for any training sample i, the positive samples of other training samples j are also used as negative samples of training sample i, and form a negative sample pair with training sample i, and the similarity of the positive sample pair and the similarity of the negative sample pair are calculated. According to the similarity of the positive sample pair and the similarity of the negative sample pair, the second contrastive learning loss is calculated.

[0131] Specifically, use s i represents the feature representation of the training sample i (i.e. the i-th training sample) after characterization, represents the feature representation of the positive sample of the training sample i after characterization, Represents the feature representation of the jth negative sample of the training sample i after characterization. Wherein, j is 1, 2, ..., K, K is a positive integer, and K is the number of negative samples of the training sample i. In this step, the following formula (3) can be used to calculate the second contrastive learning loss:

[0132]

[0133] Among them, L′ cl represents the second contrastive learning loss. Indicates i and The similarity between Indicates i and The similarity between . Indicates i and exp represents the exponential function with the natural constant e as the base. l is the number of samples participating in the training, that is, the batch size sampled during the iterative training process.

[0134] The second contrastive learning loss in formula (3) is positively correlated with the first mutual information and negatively correlated with the second mutual information. By maximizing the second contrastive learning loss, the first mutual information can be increased and the second mutual information can be reduced.

[0135] In another optional implementation of this step, the training sample and its positive sample can be combined into a positive sample pair, and the training sample and its negative sample can be combined into a negative sample pair, and the similarity of the positive sample pair and the similarity of the negative sample pair are calculated. Based on the similarity of the positive sample pair and the similarity of the negative sample pair, the second contrastive learning loss is calculated.

[0136] Specifically, in this step, the following formula (4) can be used to calculate the second contrastive learning loss:

[0137]

[0138] The meaning of each variable in formula (4) is consistent with the meaning of the same variable in the previous formula. For details, please refer to the above-mentioned related content and will not be repeated here. The second contrastive learning loss in formula (4) is positively correlated with the first mutual information and negatively correlated with the second mutual information. By maximizing the second contrastive learning loss, the first mutual information can be increased and the second mutual information can be reduced.

[0139] Optionally, the similarity between the feature representations of the training sample and its positive / negative samples may be the cosine similarity of the two feature representations, or may be calculated using other similarity calculation methods between vectors, such as Euclidean distance, Manhattan distance, etc., which is not specifically limited in this embodiment.

[0140] Step S5032: Calculate the second correlation information between the feature representation and the vector representation of the training sample as an estimated value of the mutual information between the feature representation and the vector representation of the training sample.

[0141] In this embodiment, the upper bound of the mutual information between the vector representation of the training sample before characterization and the feature representation after characterization is estimated by calculating the correlation information between the vector representation of the training sample before characterization and the feature representation after characterization, which is used as the estimated value of the mutual information between the vector representation of the training sample before characterization and the feature representation after characterization.

[0142] In an optional implementation of this step, based on the Hilbert-Schmidt independence criterion HSIC, the HSIC values ​​of the vector representation and feature representation of the training sample are calculated as the estimated value of the mutual information between the feature representation and the vector representation of the training sample. Exemplarily, the HSIC values ​​of the vector representation X and the feature representation S (i.e., the first feature representation) of the training sample can be calculated using the above formula (2). For details, refer to the relevant content in the above embodiment, which will not be repeated here.

[0143] In another implementation of this step, other methods of calculating the correlation between two sets of vector representations, such as calculating the correlation coefficient, can be used to calculate the second correlation information between the vector representation and feature representation of the training sample. This is not limited to the HSIC-based method.

[0144] Step S5033: Calculate a second loss function according to the second contrastive learning loss and the second correlation information, where the second loss function is used to increase the second contrastive learning loss and reduce the second correlation information.

[0145] In this step, a second loss function is constructed according to the second contrastive learning loss and the second correlation information, so as to maximize the second contrastive learning loss by minimizing the second loss function and minimize the second correlation information. The second loss function can be used as follows: 2 =-(L′ cl -β 2 HSIC(X,S)). Where β 2 is a hyperparameter, which can be configured and adjusted according to the needs of the actual application scenario. 2 The value is a positive number, such as β 2 It can be 1.0, etc., and is not specifically limited here.

[0146] The method of this embodiment calculates the second contrastive learning loss based on the vector representation of the training sample before characterization and the feature representation after characterization, as well as the feature representation of the positive and negative samples of the pre-labeled training sample. The second contrastive learning loss is positively correlated with the first mutual information between the feature representation of the training sample and its positive sample, and negatively correlated with the second mutual information between the feature representation of the training sample and its negative sample. The upper bound of the third mutual information between the feature representation and the vector representation of the training sample is estimated by calculating the HSIC value between the vector representation and the feature representation of the training sample, and a second loss function including the second contrastive learning loss and the HSIC value is constructed. By minimizing the second loss function, the training goals of increasing the first mutual information, reducing the second mutual information, and reducing the third mutual information can be achieved, so that the information representation model can learn important feature information while avoiding learning unnecessary feature information, which can reduce the risk of overfitting, improve the representation ability, performance and generalization ability of the information representation model after training, and thus improve the quality of information representation.

[0147] For example, Figure 7 The framework diagram of the fine-tuning training of the information representation model provided in this embodiment is as follows: Figure 7 As shown in the figure, the vector representation X of the training sample and its positive and negative samples before characterization are respectively represented by the information representation model to obtain the feature representation s after the training sample is represented. i , and the feature representation after positive and negative sample characterization Based on the feature representation of the training sample and its positive and negative samples, calculate the second contrastive learning loss L′ cl Based on the feature representation S after characterization by the information representation model and the vector representation X of the training sample before characterization, the HSIC value L of the two is calculated. hsic , as the estimated value of the lower bound of the mutual information between the two. Further, the second loss function of fine-tuning training can be: L 2 =-(L′ cl -β 2 L hsic ), β 2 is a hyperparameter and can be set to 1.0. By minimizing the second loss function L 2 , to maximize the second contrastive learning loss L′ cl , and minimize the HSIC value L hsic , thereby increasing the first mutual information between the feature representation of the training sample and its positive sample, reducing the second mutual information between the feature representation of the training sample and its negative sample, and reducing the feature representation of the training sample and the vector representation

[0148] In an optional embodiment, after obtaining the fine-tuned trained information representation model, the server may provide the information representation model to the end-side device. Exemplarily, in the case where the end-side device stores the structure of the information representation model, the server may send the parameters of the fine-tuned trained information representation model to the end-side device. The end-side device initializes the parameters of the locally stored information representation model according to the parameters of the fine-tuned trained information representation model to obtain the fine-tuned trained information representation model. Exemplarily, the server may also send download information of the fine-tuned trained information representation model to the end-side device. The end-side device downloads the fine-tuned trained information representation model to the local device based on the received download information.

[0149] In this embodiment, the server provides the information representation model obtained through fine-tuning training to the terminal side device, so that the terminal side device can store the fine-tuning trained information representation model locally. The terminal side device can use the fine-tuning trained information representation model locally to implement the information representation function, thereby improving the quality of information representation.

[0150] In an optional embodiment, after obtaining the fine-tuned and trained information representation model, the server may provide information representation services to the terminal device based on the fine-tuned and trained information representation model.

[0151] Exemplarily, when the end-side device needs to represent information, it sends an information representation request to the server, and the information representation request includes the input information to be represented. The server receives the information representation request sent by the end-side device, obtains the input information to be represented, inputs the input information into the fine-tuned trained information representation model for representation, and obtains the feature representation of the input information. The server returns the feature representation of the input information to the end-side device. The end-side device receives the feature representation of the input information returned by the server, and executes subsequent processing logic based on the feature representation of the input information.

[0152] Exemplarily, the server may also provide the end-side device with an API of the fine-tuned trained information representation model. When information representation is required, the end-side device sends an API call request to the server, and the API call request contains the input information to be represented. When the server responds to the API call request, obtains the input information to be represented, inputs the input information into the fine-tuned trained information representation model for characterization, and obtains a feature representation of the input information. The server returns the feature representation of the input information to the end-side device. The end-side device receives the feature representation of the input information returned by the server, and executes subsequent processing logic based on the feature representation of the input information.

[0153] In this embodiment, the server can provide information representation capabilities to the external terminal device based on the information representation model obtained through fine-tuning training, thereby improving the quality of information representation.

[0154] Figure 8 An exemplary flow chart of a two-stage model distillation method provided for an exemplary embodiment of the present application is as follows: Figure 8 As shown, the specific steps of this method are as follows:

[0155] Step S801, obtain a teacher model, a student model and an unsupervised training set.

[0156] Step S802: Based on the teacher model and the unsupervised training set, the student model is distilled and trained to obtain a pre-trained student model.

[0157] In this embodiment, in the first stage of model distillation, through steps S801-S802, the teacher model (reference model) and a large-scale unsupervised training set are used to perform distillation training on the student model (information representation model to be trained) to obtain a pre-trained student model. For the detailed process of the distillation training, please refer to the relevant content of any of the aforementioned model distillation method embodiments, which will not be repeated here in this embodiment.

[0158] Through the first stage of distillation training, the mutual information between the feature representations output by the teacher model and the student model is maximized, while the mutual information between the feature representation after characterization by the student model and the input data is reduced, thereby effectively solving the overfitting problem of the student model in the distillation process. The student model learns the representation ability of the teacher model by using a large-scale unsupervised training set.

[0159] Step S803: Obtain a supervised labeled data set.

[0160] Step S804: Use the supervised labeled data set to fine-tune the student model to obtain a trained student model.

[0161] In this embodiment, in the second stage of model distillation, through steps S803-S804, for the pre-trained student model obtained by distillation training, a high-quality labeled data set is used to fine-tune the student model to obtain a trained student model. For the detailed process of fine-tuning training, please refer to the relevant content of fine-tuning training of the pre-trained information representation (student model) in any of the aforementioned method embodiments, which will not be repeated here in this embodiment. Through the fine-tuning training in the second stage, the representation ability of the student model is further enhanced by using high-quality supervised data, thereby effectively narrowing the performance gap between the student model and the teacher model.

[0162] In this embodiment, through two-stage model distillation, large-scale unsupervised data and high-quality supervised data are fully utilized, thereby effectively narrowing the performance gap between the student model and the teacher model.

[0163] Fig. 9 A flowchart of an information representation method provided for an exemplary embodiment of the present application, such as Fig. 9 As shown, the specific steps of the method are as follows:

[0164] Step S901: An information representation request sent by a receiving device includes input information to be represented, and the input information includes at least one type of information among text, voice, image, and video.

[0165] Exemplarily, when the terminal device needs to represent information, it sends an information representation request to the server, and the information representation request includes the input information to be represented. The server receives the information representation request sent by the terminal device and obtains the input information to be represented.

[0166] Step S902: input the input information into the information representation model for characterization to obtain a feature representation of the input information.

[0167] After obtaining the input information to be represented, the server inputs the input information into the fine-tuned trained information representation model for representation, thereby obtaining a feature representation of the input information.

[0168] Among them, the information representation model is obtained by training through the method of any of the aforementioned method embodiments. Exemplarily, the information representation model can be obtained by model distillation training to obtain a pre-trained information representation model in the following manner: obtain a reference model and a training set for training, input samples in the training set into the information representation model and the reference model respectively, convert the samples into vector representations through the information representation model, characterize the samples based on the vector representation of the samples to obtain the first feature representation of the samples, and characterize the samples through the reference model to obtain the second feature representation of the samples; according to the vector representation, the first feature representation and the second feature representation of the samples, the information representation model is trained with the goal of maximizing the correlation between the first feature representation and the second feature representation of the samples and minimizing the correlation between the first feature representation and the vector representation of the samples. For the specific distillation training process, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0169] Exemplarily, the information representation model may also be a trained information representation model obtained by fine-tuning the pre-trained information representation model obtained by distillation training. The process of fine-tuning training refers to the relevant content of the aforementioned embodiment and will not be repeated here.

[0170] Step S903: Return the feature representation of the input information to the terminal device.

[0171] After obtaining the feature representation of the input information, the server returns the feature representation of the input information to the terminal device. The terminal device receives the feature representation of the input information returned by the server and executes subsequent processing logic based on the feature representation of the input information.

[0172] In this embodiment, information representation is performed based on a lightweight information representation model obtained through distillation training, thereby improving information representation efficiency while ensuring information representation quality.

[0173] The following takes the distillation of the text representation model as an example to illustrate the above-mentioned model distillation method. Fig.10 A flow chart of a text representation model distillation method provided for an exemplary embodiment of the present application. Fig.10 As shown, the specific steps of this method are as follows:

[0174] Step S1001: Obtain a pre-trained model for text representation, a text representation model to be trained, and a first text set.

[0175] Among them, the pre-trained model used for text representation can be BERT, a pre-trained language model, a large-scale model, a multimodal pre-trained model, etc., which is not specifically limited in this embodiment.

[0176] The text representation model to be trained is a lightweight model with the same task capability as the pre-trained model for text representation, and has fewer parameters than the pre-trained model. The text representation model can be obtained by compressing the pre-trained model using model compression methods such as pruning, quantization, low-rank decomposition, and parameter sharing, or it can be a reconstructed model, which is not specifically limited in this embodiment.

[0177] The first text set includes a large-scale unannotated text corpus, which may come from a public text corpus, a public data set, or may be collected by a server, or may be provided by a user through a terminal device, which is not specifically limited in this embodiment.

[0178] Step S1002: input the text in the first text set into the text representation model and the pre-trained model respectively, convert the text into a vector representation through the text representation model, perform text representation based on the vector representation of the text to obtain a first feature representation of the text, and characterize the text through the pre-trained model to obtain a second feature representation of the text.

[0179] In this step, the pre-trained model is used as the reference model, and the training set used is the first text set. The texts in the first text set are the samples in the training set. It is implemented in a manner similar to the aforementioned step S201. For the specific implementation method, please refer to the relevant content in the aforementioned embodiments, and this embodiment is not specifically limited here.

[0180] Step S1003: According to the vector representation, the first feature representation and the second feature representation of the text, the text representation model is trained to obtain a pre-trained text representation model by maximizing the correlation between the first feature representation and the second feature representation of the text and minimizing the correlation between the first feature representation and the vector representation of the text as the training objectives.

[0181] The correlation between the first feature representation and the second feature representation of the text can be measured using the mutual information between the first feature representation and the second feature representation of the text. Of course, other information can also be used to measure the correlation between the first feature representation and the second feature representation of the text. The correlation between the first feature representation and the vector representation of the text can be measured using the mutual information between the first feature representation and the vector representation of the text. Of course, other information can also be used to measure the correlation between the first feature representation and the vector representation of the text.

[0182] In this embodiment, the information bottleneck theory is applied to the distillation of the text representation model. The first loss function used in the distillation training is constructed according to the mutual information between the first feature representation output by the text representation model and the second feature representation output by the pre-trained model (i.e., the teacher model), as well as the mutual information between the first feature representation of the sample after representation by the text representation model and the vector representation of the sample before representation. By minimizing the first loss function, the mutual information between the feature representation obtained by the pre-trained model and the text representation model is maximized, while the mutual information between the feature representation obtained by the text representation model and the input data is minimized. This allows the text representation model to learn important feature information in the input information while avoiding learning unnecessary feature information. This can reduce the risk of overfitting, thereby improving the text representation capability, performance, and generalization capability of the trained text representation model.

[0183] This step is consistent with the first loss function used in the aforementioned step S202. The samples in the training set in this step are texts, and the implementation method is similar to that of the aforementioned step S202. For details, please refer to the relevant content in the aforementioned method embodiment, which will not be repeated here in this embodiment.

[0184] In this embodiment, the information bottleneck theory is applied to the distillation of the text representation model. The first loss function used in the distillation training is constructed according to the mutual information between the first feature representation output by the text representation model and the second feature representation output by the pre-trained model, as well as the mutual information between the first feature representation output by the text representation model and the vector representation before representation. By minimizing the first loss function, the mutual information between the feature representations output by the pre-trained model and the text representation model is maximized, while the mutual information between the feature representation output by the text representation model and the input data is minimized. This allows the text representation model to learn important text feature information while avoiding learning unnecessary feature information. This can reduce the risk of overfitting, thereby improving the text representation capability, performance and generalization ability of the trained text representation model, thereby improving the quality of text representation.

[0185] In an optional embodiment, after obtaining a pre-trained text representation model (text representation model), the server may provide the text representation model to the end-side device. Exemplarily, when the end-side device stores the structure of the text representation model, the server may send the trained parameters of the text representation model to the end-side device. The end-side device initializes the parameters of the locally stored text representation model according to the trained parameters of the text representation model to obtain the pre-trained text representation model (text representation model).

[0186] Exemplarily, the server may also send download information of a pre-trained text representation model (text representation model) to the end-side device. The end-side device downloads the pre-trained text representation model to a local device according to the received download information to obtain the pre-trained text representation model.

[0187] In this embodiment, the server provides the pre-trained text representation model obtained through distillation training to the terminal device, so that the terminal device can store the pre-trained text representation model locally. The terminal device can use the pre-trained text representation model locally to implement the text representation function, thereby improving the quality of text representation.

[0188] In an optional embodiment, after obtaining a pre-trained text representation model (text representation model), the server may provide a text representation service to the end-side device based on the pre-trained text representation model.

[0189] Exemplarily, when the end-side device needs to perform text representation, it sends a text representation request to the server, and the text representation request includes the input text to be represented. The server receives the text representation request sent by the end-side device, obtains the input text to be represented, inputs the input text into the text representation model for representation, and obtains the feature representation of the input text. The server returns the feature representation of the input text to the end-side device. The end-side device receives the feature representation of the input text returned by the server, and executes subsequent natural language processing logic based on the feature representation of the input text.

[0190] Exemplarily, the server can also provide the end-side device with an API of a pre-trained text representation model (text representation model). When the end-side device needs to perform text representation, it sends an API call request to the server, and the API call request contains the input text to be represented. When the server responds to the API call request, it obtains the input text to be represented, inputs the input text into the text representation model for characterization, and obtains a feature representation of the input text. The server returns the feature representation of the input text to the end-side device. The end-side device receives the feature representation of the input text returned by the server, and executes subsequent natural language processing logic based on the feature representation of the input text.

[0191] In this embodiment, the server can provide text representation capabilities to the external terminal device based on the pre-trained text representation model obtained through distillation training, which can improve the quality of text representation.

[0192] Based on any of the foregoing method embodiments, in an optional embodiment, after obtaining a pre-trained text representation model (text representation model) through model distillation, the server can also use a high-quality annotated dataset to fine-tune the pre-trained text representation model (text representation model) to further improve the text representation capability and performance of the text representation model. Fig.11 A flowchart of the method for fine-tuning the text representation model provided in this embodiment, such as Fig.11 As shown, the specific steps of this method are as follows:

[0193] Step S1101: Obtain a second text set, where the training data in the second text set includes: training text, positive samples of pre-labeled training text, and multiple negative samples.

[0194] In this embodiment, the second text set used for fine-tuning the pre-trained text representation model (text representation model) is a supervised data set with pre-labeled positive and negative samples. The second text set includes not only training texts, but also pre-labeled positive and negative samples of training texts. The second text set can use a public labeled text set; it can also be obtained by collecting large-scale unlabeled text samples by the server and labeling the positive and negative samples of the text samples; it can also be provided by the user through the terminal device, which is not specifically limited in this embodiment.

[0195] Different from the first text set used in the distillation training stage, in the fine-tuning training stage, the second high-quality supervised text set with labeled positive and negative samples is used for training, which can further improve the text representation ability of the text representation model and thus improve the text representation quality.

[0196] Each piece of training data in the second text set includes three parts: a training text, a pre-labeled positive sample of the training text, and a plurality of pre-labeled negative samples of the training text.

[0197] In an optional application scenario, the second text set may be a vertical field annotated text set provided by the end-side device. The server uses the vertical field annotated text set to fine-tune the pre-trained text representation model to obtain a dedicated text representation model suitable for the vertical field. The server provides the trained vertical field dedicated text representation model to the end-side device.

[0198] In the fine-tuning training phase, the server iterates multiple rounds of fine-tuning training on the text representation model, and obtains a trained text representation model after the iteration. In one iteration of training, the server samples training data of a preset batch size (represented by l, where l is a positive integer) from the second text set, and uses the sampled batch of training data for training.

[0199] Step S1102: input the training text and the positive samples and negative samples into the pre-trained text representation model for representation, and obtain the vector representation and feature representation of the training text, and the feature representation of the positive samples and negative samples of the training text.

[0200] During an iterative training process, the server inputs the training text and its positive and negative samples in the training data used for this round of iterative training into the pre-trained text representation model, converts the training text into a vector representation and represents it through the pre-trained text representation model, and outputs the vector representation and feature representation of the training text; the positive and negative samples are converted and represented respectively through the pre-trained text representation model, and the feature representation of the positive and negative samples is output.

[0201] Step S1103, calculate a second loss function based on the vector representation and feature representation of the training text, and the feature representation of the positive text and the negative text of the training text, wherein the second loss function is used to increase the first mutual information between the feature representation of the training text and the feature representation of the positive text, reduce the second mutual information between the feature representation of the training text and the feature representation of the negative text, and reduce the third mutual information between the feature representation of the training text and the vector representation.

[0202] This step is consistent with the second loss function used in the aforementioned step S503. In this step, the training samples in the labeled data set are training texts. The second loss function is calculated in the same manner as the aforementioned step S503. For details, please refer to the relevant content in the aforementioned method embodiment, which will not be repeated here in this embodiment.

[0203] Step S1104: train the parameters of the text representation model according to the second loss function to obtain a trained text representation model.

[0204] After calculating the second loss function, the parameters of the text representation model are updated through back propagation according to the second loss function. After multiple rounds of iterative training, the trained text representation model can be obtained. Since the text representation model is fine-tuned using a high-quality supervised text set, the text representation ability of the text representation model can be further improved, thereby improving the quality of text representation.

[0205] In this embodiment, the information bottleneck theory is applied to the fine-tuning training of the text representation model (text representation model). The second loss function used in the fine-tuning training is constructed according to the mutual information between the feature representation of the training text after characterization and the feature representation of the comparison text (including positive and negative samples), and the mutual information between the feature representation of the training text after characterization and the vector representation of the training text before characterization. By minimizing the second loss function, the first mutual information between the feature representation of the training text and the feature representation of the positive sample is increased, the second mutual information between the feature representation of the training text and the feature representation of the negative sample is reduced, and the third mutual information between the feature representation of the training text and the vector representation is reduced, so that the text representation model can learn important feature information in the input text while avoiding learning unnecessary feature information, which can reduce the risk of overfitting, improve the text representation ability, performance and generalization ability of the trained text representation model, and thus improve the quality of text representation.

[0206] In an optional embodiment, after obtaining a fine-tuned text representation model (text representation model), the server may provide the fine-tuned text representation model to the end-side device. Exemplarily, in a case where the end-side device stores the structure of the text representation model, the server may send the parameters of the text representation model after fine-tuning training to the end-side device. The end-side device initializes the parameters of the locally stored text representation model according to the parameters of the text representation model after fine-tuning training, and obtains the fine-tuned text representation model (text representation model).

[0207] Exemplarily, the server may also send download information of the fine-tuned text representation model (text representation model) to the end-side device. The end-side device downloads the fine-tuned text representation model to the local device according to the received download information to obtain the fine-tuned text representation model.

[0208] In this embodiment, the server provides the text representation model obtained through fine-tuning training to the terminal side device, so that the terminal side device can store the fine-tuning trained text representation model locally. The terminal side device can use the fine-tuning trained text representation model locally to implement the text representation function, thereby improving the quality of text representation.

[0209] In an optional embodiment, after obtaining the fine-tuned and trained text representation model (text representation model), the server may provide a text representation service to the terminal device based on the fine-tuned and trained text representation model.

[0210] Exemplarily, when the end-side device needs to perform text representation, it sends a text representation request to the server, and the text representation request includes the input text to be represented. The server receives the text representation request sent by the end-side device, obtains the input text to be represented, inputs the input text into the fine-tuned trained text representation model for representation, and obtains the feature representation of the input text. The server returns the feature representation of the input text to the end-side device. The end-side device receives the feature representation of the input text returned by the server, and executes subsequent natural language processing logic based on the feature representation of the input text.

[0211] Exemplarily, the server may also provide the end-side device with an API of a fine-tuned and trained text representation model (text representation model). When text representation is required, the end-side device sends an API call request to the server, and the API call request contains the input text to be represented. When the server responds to the API call request, obtains the input text to be represented, inputs the input text into the fine-tuned and trained text representation model for characterization, and obtains a feature representation of the input text. The server returns the feature representation of the input text to the end-side device. The end-side device receives the feature representation of the input text returned by the server, and executes subsequent natural language processing logic based on the feature representation of the input text.

[0212] In this embodiment, the server can provide text representation capabilities to the external terminal device based on the text representation model obtained through fine-tuning training, thereby improving the quality of text representation.

[0213] Fig.12 A flowchart of a text representation method provided for an exemplary embodiment of the present application, such as Fig.12 As shown, the specific steps of the method are as follows:

[0214] Step S1201: A text representation request sent by a receiving device includes input text to be represented.

[0215] Exemplarily, when the terminal device needs to perform text representation, it sends a text representation request to the server, and the text representation request includes the input text to be represented. The server receives the text representation request sent by the terminal device and obtains the input text to be represented.

[0216] Step S1202: input the input text into a text representation model for characterization to obtain a feature representation of the input text.

[0217] After obtaining the input text to be represented, the server inputs the input text into the fine-tuned trained text representation model for representation to obtain a feature representation of the input text.

[0218] Wherein, the text representation model is obtained by training by the method of any of the aforementioned method embodiments. Exemplarily, the text representation model can be a pre-trained text representation model obtained by model distillation training in the following manner: obtain a pre-trained model for text representation, a text representation model to be trained, and a first text set; input the text in the first text set into the text representation model and the pre-trained model respectively, convert the text into a vector representation through the text representation model, perform text representation based on the vector representation of the text, obtain a first feature representation of the text, and characterize the text through the pre-trained model to obtain a second feature representation of the text; according to the vector representation, the first feature representation, and the second feature representation of the text, maximize the mutual information between the first feature representation and the second feature representation of the text, and minimize the mutual information between the first feature representation and the vector representation of the text as the training goal, and train the text representation model. For the specific distillation training process, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here.

[0219] Exemplarily, the text representation model may also be a trained text representation model obtained by fine-tuning the pre-trained text representation model obtained by distillation training. The process of fine-tuning training refers to the relevant content of the aforementioned embodiment and will not be repeated here.

[0220] Step S1203: Return the feature representation of the input text to the terminal device.

[0221] After obtaining the feature representation of the input text, the server returns the feature representation of the input text to the terminal device. The terminal device receives the feature representation of the input text returned by the server and executes subsequent processing logic based on the feature representation of the input text.

[0222] In this embodiment, text representation is performed based on a lightweight text representation model obtained through distillation training, thereby improving the efficiency of text representation while ensuring the quality of text representation.

[0223] Fig.13 A schematic diagram of the structure of a server provided in an embodiment of the present application. Fig.13 As shown, the server of this embodiment may include: at least one processor 1301; and a memory 1302 connected to the at least one processor in communication; wherein the memory 1302 stores instructions executable by the at least one processor 1301, and the instructions are executed by the at least one processor 1301, so that the server executes the method of any of the above embodiments. The implementation principle and technical effects of the server provided in this embodiment can be referred to the above embodiments, and will not be repeated here.

[0224] Optionally, the memory 1302 may be independent or integrated with the processor 1301. Fig.13As shown, the server also includes: a firewall 1303, a load balancer 1304, a communication component 1305, a power supply component 1306 and other components. Fig.13 Only some components are shown schematically, which does not mean that the server only includes Fig.13 Components shown.

[0225] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method of any of the aforementioned embodiments is implemented. The specific functions and technical effects that can be achieved are not repeated here.

[0226] The present application also provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server executes the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0227] The embodiment of the present application provides a chip, including: a processing module and a communication interface, the processing module can execute the technical solution of the server in the aforementioned method embodiment. Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution provided by any of the aforementioned method embodiments.

[0228] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0229] It should be understood that the above-mentioned processor can be a processing unit (Central Processing Unit, referred to as CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as DSP), application-specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0230] The above-mentioned memory can be an object storage service (OSS). The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second generation mobile communication system (2G), a third generation mobile communication system (3G), a fourth generation mobile communication system (4G) / long term evolution (LTE), a fifth generation mobile communication system (5G) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies. The above-mentioned power supply component provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing and distributing power for the device where the power supply component is located.

[0231] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0232] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0233] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0234] The order of the above-mentioned embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. In addition, in some of the processes described in the above-mentioned embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed or executed in parallel in the order in which they appear in this article, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0235] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0236] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0237] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for processing an information representation model, characterized in that: include: Obtaining an information representation model to be trained, as well as a reference model and a training set for training; Input samples in the training set into the information representation model and the reference model respectively, convert the samples into vector representations through the information representation model, characterize the samples based on the vector representations of the samples to obtain a first feature representation of the samples, and characterize the samples through the reference model to obtain a second feature representation of the samples; According to the vector representation, the first feature representation and the second feature representation of the sample, the information representation model is trained with the training objectives of maximizing the correlation between the first feature representation and the second feature representation of the sample and minimizing the correlation between the first feature representation and the vector representation of the sample to obtain a pre-trained information representation model.

2. The method according to claim 1, characterized in that The method of training the information representation model based on the vector representation, the first feature representation and the second feature representation of the sample, taking maximizing the correlation between the first feature representation and the second feature representation of the sample and minimizing the correlation between the first feature representation and the vector representation of the sample as training objectives, and obtaining a pre-trained information representation model comprises: Calculating a first contrastive learning loss based on the first feature representation and the second feature representation of the sample as an estimated value of the mutual information between the first feature representation and the second feature representation of the sample; Calculating first correlation information between the vector representation and the first feature representation of the sample as an estimated value of mutual information between the first feature representation and the vector representation of the sample; Calculating a first loss function according to the first contrastive learning loss and the first correlation information, wherein the first loss function is used to increase the first contrastive learning loss and reduce the first correlation information; According to the first loss function, the parameters of the information representation model are trained.

3. The method according to claim 2, characterized in that The calculating the contrastive learning loss according to the first feature representation and the second feature representation of the sample includes: For any sample i, the first feature representation of sample i and the second feature representation form a positive sample pair, and the first feature representation of sample i and the second feature representation of any sample j form a negative sample pair, where sample j and sample i are different samples; Calculate the similarity of positive sample pairs and the similarity of negative sample pairs; A first contrastive learning loss is calculated according to the similarity of the positive sample pair and the similarity of the negative sample pair.

4. The method according to claim 2, characterized in that: The calculating the first correlation information between the vector representation and the first feature representation of the sample as an estimated value of the mutual information between the first feature representation and the vector representation of the sample includes: Based on the Hilbert-Schmidt independence criterion HSIC, the HSIC values ​​of the vector representation and the first feature representation of the sample are calculated as an estimated value of the mutual information between the first feature representation and the vector representation of the sample.

5. The method according to claim 1, characterized in that After obtaining the pre-trained information representation model, it also includes: Acquire a labeled data set, wherein the training data in the labeled data set includes: training samples, positive samples and negative samples of the training samples that are pre-labeled; Inputting the training samples and the positive samples and negative samples into the pre-trained information representation model for conversion and characterization, respectively, to obtain vector representation and feature representation of the training samples, as well as feature representation of the positive samples and negative samples of the training samples; Calculate a second loss function according to the vector representation and feature representation of the training sample, and the feature representation of the positive sample and the negative sample of the training sample, wherein the second loss function is used to increase the first mutual information between the feature representation of the training sample and the feature representation of the positive sample, reduce the second mutual information between the feature representation of the training sample and the feature representation of the negative sample, and reduce the third mutual information between the feature representation of the training sample and the vector representation; According to the second loss function, the parameters of the information representation model are trained to obtain a trained information representation model.

6. The method according to claim 5, characterized in that The calculating a second loss function according to the vector representation and feature representation of the training sample and the feature representation of the positive sample and the negative sample of the training sample comprises: Calculating a second contrastive learning loss according to the feature representation of the training sample and the feature representation of the positive sample and the negative sample of the training sample, wherein the second contrastive learning loss is positively correlated with the first mutual information and negatively correlated with the second mutual information; Calculating second correlation information between the feature representation and the vector representation of the training sample as an estimated value of mutual information between the feature representation and the vector representation of the training sample; The second loss function is calculated according to the second contrastive learning loss and the second correlation information, and the second loss function is used to increase the second contrastive learning loss and reduce the second correlation information.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: Acquire a pre-trained model for information representation specified by the end-side device, and use the pre-trained model as a reference model; The pre-trained model is compressed to obtain an information representation model.

8. The method according to claim 7, characterized in that Also includes: Sending the information to the end-side device represents the parameters of the model after training; or, The information representing the download information of the model is sent to the terminal side device.

9. An information representation method, characterized in that: include: An information representation request sent by a receiving terminal device, wherein the information representation request includes input information to be represented, and the input information includes at least one type of information of text, voice, image, and video; Inputting the input information into the information representation model for characterization to obtain a feature representation of the input information, wherein the information representation model is trained in the following manner: obtaining a reference model and a training set for training, inputting samples in the training set into the information representation model and the reference model respectively, converting the samples into vector representations through the information representation model, characterizing the samples based on the vector representations of the samples to obtain a first feature representation of the samples, and characterizing the samples through the reference model to obtain a second feature representation of the samples; According to the vector representation, the first feature representation and the second feature representation of the sample, the information representation model is trained by maximizing the correlation between the first feature representation and the second feature representation of the sample and minimizing the correlation between the first feature representation and the vector representation of the sample as training objectives; Return a feature representation of the input information to the terminal device.

10. A method for processing an information representation model, characterized in that: include: Obtaining a pre-trained model for text representation, a text representation model to be trained, and a first text set; Inputting the texts in the first text set into the text representation model and the pre-trained model respectively, converting the texts into vector representations through the text representation model, performing text representation based on the vector representations of the texts to obtain a first feature representation of the texts, and representing the texts through the pre-trained model to obtain a second feature representation of the texts; According to the vector representation, the first feature representation and the second feature representation of the text, the text representation model is trained with the training objectives of maximizing the correlation between the first feature representation and the second feature representation of the text and minimizing the correlation between the first feature representation and the vector representation of the text to obtain a pre-trained text representation model.

11. The method according to claim 10, characterized in that After obtaining the pre-trained text representation model, it also includes: Acquire a second text set, wherein the training data in the second text set includes: training text, and pre-labeled positive samples and negative samples of the training text; Inputting the training text and the positive samples and negative samples into a pre-trained text representation model for characterization, respectively, to obtain a vector representation and a feature representation of the training text, as well as a feature representation of the positive samples and negative samples of the training text; Calculating a second loss function based on the vector representation and feature representation of the training text, and the feature representation of the positive text and the negative text of the training text, wherein the second loss function is used to increase the first mutual information between the feature representation of the training text and the feature representation of the positive text, reduce the second mutual information between the feature representation of the training text and the feature representation of the negative text, and reduce the third mutual information between the feature representation of the training text and the vector representation; According to the second loss function, the parameters of the text representation model are trained to obtain a trained text representation model.

12. An information representation method, characterized in that: include: A text representation request sent by a receiving terminal device, wherein the text representation request includes input text to be represented; Input the input text into the text representation model for representation to obtain a feature representation of the input text, wherein the text representation model is trained in the following manner: obtain a pre-trained model for text representation, a text representation model to be trained, and a first text set; input text in the first text set into the text representation model and the pre-trained model respectively, convert the text into a vector representation through the text representation model, perform text representation based on the vector representation of the text to obtain a first feature representation of the text, and characterize the text through the pre-trained model to obtain a second feature representation of the text; train the text representation model based on the vector representation, the first feature representation, and the second feature representation of the text, with the training objectives of maximizing the correlation between the first feature representation and the second feature representation of the text and minimizing the correlation between the first feature representation and the vector representation of the text; Return the feature representation of the input text to the terminal device.

13. A server, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the server to execute the method described in any one of claims 1-12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

Images