Model training method and related apparatus

By using the content description model in the AI ​​model to extract text features and multimedia features and updating the model based on the difference, the problem of small parameters on resource-constrained devices is solved, and the multimedia data processing accuracy is improved.

WO2024234706A1PCT designated stage expired Publication Date: 2024-11-21HUAWEI TECH CO LTD

Patent Information

Application Number
PCT/CN2024/072229
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-16
Filing Date
2024-01-15
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

In the existing AI model, the small amount of parameters makes it difficult to regulate performance when the parameters are limited, and when the parameters are large, the resource consumption is too much, making it difficult to improve performance when the parameters are limited.

Method used

By entering multimedia data into content description model to obtain target text, extract text features and multimedia features, and update the model based on loss function values ​​to minimize the difference between text features and multimedia features, thereby improving the model's multimedia data processing accuracy .

Benefits of technology

It effectively improves the accuracy of the AI ​​model when processing multimedia data, makes full use of the information of the content description model, enhances the model's cognitive ability of the multimedia data, and is suitable for devices with resource limitations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024072229_21112024_PF_FP_ABST
    Figure CN2024072229_21112024_PF_FP_ABST
Patent Text Reader

Abstract

A model training method, which is applied to the technical field of artificial intelligence. In the method, multimedia data, which serves as training data, is first input into a content description model, the content description model outputs text for describing the content in the multimedia data, and the text is subjected to feature extraction, such that a text feature is obtained; the multimedia data is then input into a model to be trained, and the model to be trained performs extraction to obtain a multimedia feature; and a loss function is then constructed on the basis of the text feature and multimedia feature that correspond to the multimedia data, such that the model to be trained is trained. The content description model outputs description text for the multimedia data, such that more detailed information in the multimedia data can be provided; and the loss function is constructed on the basis of the text feature and the multimedia feature, such that matching between the text feature and the multimedia feature can be realized, thereby facilitating enhancing the cognition of a model for the multimedia data, and thus effectively improving the precision of the model for processing the multimedia data.
Need to check novelty before this filing date? Find Prior Art

Description

A model training method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 16, 2023, with application number 202310552276.X and invention name “A model training method and related devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a model training method and related devices. Background Art

[0003] With the development of AI technology, neural network-based AI models have made significant progress in recent years in processing and analyzing various media signals, including images, videos, text, and speech. In the field of computer vision, for example, AI models are often used to perform image processing tasks such as image classification, image segmentation, image recognition, and object detection, achieving excellent performance.

[0004] Generally speaking, the performance of an AI model is closely related to the number of its parameters. The larger the number of parameters, the easier it is to tune the model's performance. On the other hand, if the number of parameters is too small, it is often difficult to tune the model's performance well.

[0005] However, the larger the number of parameters in an AI model, the more processing resources it requires during operation. For some devices with limited processing resources, such as smartphones or self-driving cars, these devices are often only able to deploy AI models with smaller number of parameters.

[0006] Therefore, how to improve the performance of AI models as much as possible when the number of AI model parameters is limited has become an urgent problem to be solved.

[0007] Summary of the Invention

[0008] This application provides a model training method that can improve the accuracy of the model in processing multimedia data.

[0009] In a first aspect, the present application provides a model training method for use in the field of AI technology. The method comprises: first, inputting multimedia data in a training dataset into a content description model to obtain a target text output by the content description model. The multimedia data may be, for example, an image, video, or audio, and the target text output by the content description model is used to describe the content in the multimedia data. For example, assuming the multimedia data is an image, the target text may specifically be "a puppy running on the grass."

[0010] Then, the features of the target text are extracted through AI models or other means to obtain the first text features. Furthermore, the multimedia data is input into the first model to obtain the first multimedia features extracted by the first model. In other words, the first text features and the first multimedia features are both obtained based on the same multimedia data, and the first multimedia features are obtained based on the first model, while the first text features are obtained based on other AI models. Furthermore, when the multimedia data is an image, the first multimedia features extracted by the first model are image features; when the multimedia data is a video, the first multimedia features extracted by the first model are video features; and when the multimedia data is voice, the first multimedia features extracted by the first model are voice features.

[0011] Finally, based on the loss function value, the first model is updated to obtain the second model. The loss function value is obtained based on the difference between the first text feature and the first multimedia feature. During the training process of the first model, the training goal of the first model is to reduce the loss function value corresponding to the first model. Since the loss function value is obtained based on the difference between the first text feature and the first multimedia feature, the training goal of the first model is actually to reduce the difference between the first text feature and the first multimedia feature, that is, to make the multimedia features extracted by the first model as close as possible to the text features corresponding to the text output by the content description model. In this way, by constructing a loss function based on the difference between the first text feature and the first multimedia feature, the rich information in the multimedia data identified by the content description model can be used to help the first model realize the recognition of multimedia data, and the content description model with higher performance can be fully utilized to enhance the accuracy of the first model in multimedia data processing tasks.

[0012] In this approach, multimedia data serving as training data is first input into a content description model. The content description model then outputs text describing the content in the multimedia data and performs feature extraction on this text to obtain text features. This multimedia data is then input into the model to be trained, which extracts multimedia data features. A loss function is then constructed based on the text features and multimedia features corresponding to the multimedia data to train the model to be trained. By having the content description model output descriptive text specific to the multimedia data, more detailed information about the multimedia data is provided. Furthermore, constructing a loss function based on text features and multimedia features allows for a match between the text features and multimedia features, helping to enhance the model's understanding of the multimedia data and effectively improving the model's accuracy in processing multimedia data.

[0013] In one possible implementation, the content description model is a multimodal model, which can process different types of input data, such as images, videos, texts, or voices.

[0014] When the content description model is a multimodal model, multimedia data and at least one instruction can be input into the content description model to obtain the target text output by the content description model, wherein the at least one instruction is used to instruct the content description model to describe the content in the multimedia data.

[0015] Since the content description model is a multimodal model, it can receive data in multiple different modalities as input data and process data in different modalities. Therefore, the content description model can recognize the content indicated by the text instructions or voice instructions, and then recognize the multimedia data based on the content indicated by the text instructions or voice instructions and output the target text used to describe the content in the multimedia data.

[0016] Simply put, this solution actually uses the content description model as a visual question-answering system. By inputting multimedia data and custom questions (i.e., text instructions or voice instructions) into the content description model, the target text output by the content description model that describes the content of the multimedia data is obtained, ensuring that text that can accurately describe the content of the multimedia data can be obtained.

[0017] In one possible implementation, when multiple instructions are input into a content description model, the multimedia data and the multiple instructions can be first input into the content description model to obtain multiple texts output by the content description model. The content description model then performs a fusion process on the multiple texts to obtain the target text. Each of the multiple instructions instructs the content description model to describe the content of the multimedia data, and different instructions instruct the content description model to describe different content. Thus, the multiple instructions correspond one-to-one to the multiple texts.

[0018] In this solution, because different instructions instruct the content description model to describe different content within the multimedia data (i.e., instruct the content description model), the different texts in the multiple text output by the content description model also describe different content within the multimedia data. This allows the content description model to fuse these multiple texts, thereby outputting target texts that describe the multimedia data content from different perspectives. This results in target texts containing more detailed information about the multimedia data, facilitating subsequent matching of the target text's features with those of the multimedia data to construct an accurate loss function.

[0019] In one possible implementation, the process of inputting multimedia data and multiple instructions into a content description model may specifically include: inputting the multimedia data and a first instruction into the content description model to obtain a first text output by the content description model; then, inputting the multimedia data and a second instruction into the content description model to obtain a second text output by the content description model.

[0020] The multiple instructions include a first instruction and a second instruction, the multiple texts include a first text and a second text, and the second instruction is generated based on the first text.

[0021] In this solution, after a combination of an instruction and multimedia data is input into a content description model and the text output by the content description model is obtained, the next instruction that needs to be input into the content description model can be generated based on the text output by the content description model, so that the content description model can more accurately understand the content to be described, and thus output text that can accurately describe the content of the multimedia data.

[0022] In a possible implementation, the at least one instruction input into the content description model is a text instruction or a voice instruction.

[0023] In one possible implementation, in the process of determining the loss function value, the first text feature and the first multimedia feature are first normalized to obtain a second text feature and a second multimedia feature located in the same feature space, so as to facilitate comparison between the text feature and the multimedia feature.

[0024] Then, the text features and multimedia features corresponding to each multimedia data in the other multiple multimedia data in the training data set are obtained, wherein the method for obtaining the text features corresponding to each multimedia data is the same as the method for obtaining the second text features, and the method for obtaining the multimedia features corresponding to each multimedia data is the same as the method for obtaining the second multimedia features, and the multimedia data is located in the training data set. In other words, for each multimedia data in the training data set, the description text corresponding to each multimedia data is first output through the content description model, and the text features of the description text corresponding to each multimedia data are output based on the text feature extraction model, and the text features output by the text feature extraction model are normalized to obtain the text features corresponding to each multimedia data in the training data set; secondly, feature extraction processing is performed on each multimedia data through the first model, and the extracted multimedia features are normalized to obtain the multimedia features corresponding to each multimedia data.

[0025] Finally, based on the multiple first relationships and the multiple second relationships, a difference between the first text feature and the first multimedia feature is determined. The multiple first relationships include a size relationship between the second text feature and a text feature corresponding to each multimedia data item, and the multiple second relationships include a size relationship between the second multimedia feature and a multimedia feature corresponding to each multimedia data item. The multiple first relationships correspond to the multiple second relationships in a one-to-one manner, and the number of target relationship pairs is negatively correlated with the difference. The target relationship pairs include a first relationship and a second relationship that have the same size relationship and correspond to each other.

[0026] In simple terms, in this embodiment, the difference between the second text feature and the second multimedia feature is not directly compared. Instead, the second text feature is compared with other text features to determine the size ranking relationship between the second text feature and multiple text features, and the second multimedia feature is compared with other multimedia features to determine the size ranking relationship between the second multimedia feature and multiple multimedia features. If the size ranking relationship between multiple multimedia features is closer to the size ranking relationship between multiple text features, it means that the multimedia features extracted by the first model are closer to the text features extracted by the text feature extraction model, and therefore the loss function value is smaller.

[0027] In this scheme, by constructing a ranking-based loss function to characterize the differences between text features and multimedia features, it is possible to cleverly achieve comparison between features of different dimensions and improve the feasibility of the scheme.

[0028] In one possible implementation, the first multimedia feature is an intermediate feature extracted by the first model. In the process of constructing the loss function, a data processing result can be obtained, which is output by the first model after processing the multimedia data. In addition, based on the data processing result and the true label of the multimedia data, a first difference value is determined, and the first difference value is used to represent the difference between the data processing result and the true label of the multimedia data. In addition, the loss function value used to update the first model is obtained based on the first difference value and the second difference value, and the second difference value is obtained based on the difference between the first text feature and the first multimedia feature.

[0029] In this solution, the total loss function used to train the first model can be composed of two loss functions, one of which is constructed based on the multimedia features extracted by the first model and the text features extracted by the text feature extraction model, and the other loss function is constructed based on the data processing results output by the multimedia data and the true labels of the training data. In this way, by using the total loss function constructed based on the above two loss functions to update the first model, the training of the first model can be guided from two perspectives: learning the detailed information of the multimedia data and learning the final optimization direction, further improving the multimedia data processing accuracy of the first model.

[0030] In one possible implementation, the first text feature is extracted by a text feature extraction model. The text feature extraction model is a pre-trained model, or the text feature extraction model is a model trained together with the first model, and the loss function used to train the text feature extraction model is the same as the loss function used to train the first model.

[0031] In one possible implementation, the content description model is a pre-trained model. Alternatively, the content description model is a model trained together with the first model, and the loss function used to train the content description model is the same as the loss function used to train the first model.

[0032] In one possible implementation, the content description model has a larger number of parameters than the first model. Using a content description model with a larger number of parameters to output text describing multimedia data can effectively improve the accuracy of the output text and ensure the accuracy of the first model obtained through subsequent training.

[0033] In one possible implementation, when the multimedia data is an image, the first model is used to perform any one of the following multimedia data processing tasks: image classification, image segmentation, image recognition, object detection, image generation, and image enhancement;

[0034] Alternatively, when the multimedia data is a video, the first model is used to perform any one of the following multimedia data processing tasks: video classification, video behavior detection, and video generation;

[0035] Alternatively, when the multimedia data is speech, the first model is used to perform any one of the following multimedia data processing tasks: speech recognition and speech synthesis.

[0036] The second aspect of the present application provides a model training device, including: a text acquisition module, used to input multimedia data into a content description model to obtain a target text output by the content description model, and the target text is used to describe the content in the multimedia data; a processing module, used to extract features of the target text to obtain a first text feature; the processing module is also used to input multimedia data into the first model to obtain a first multimedia feature extracted by the first model; the processing module is also used to update the first model based on a loss function value to obtain a second model, wherein the loss function value is obtained based on the difference between the first text feature and the first multimedia feature.

[0037] In one possible implementation, the content description model is a multimodal model that can process different types of input data; the text acquisition module is further used to input multimedia data and at least one instruction into the content description model to obtain the target text output by the content description model, wherein the at least one instruction is used to instruct the content description model to describe the content in the multimedia data.

[0038] In one possible implementation, when at least one instruction is multiple instructions, the text acquisition module is further used to: input multimedia data and multiple instructions into a content description model to obtain multiple texts output by the content description model, wherein each of the multiple instructions is used to instruct the content description model to describe the content in the multimedia data, and different instructions instruct the content description model to describe different contents, and the multiple instructions correspond to multiple texts; and perform fusion processing on the multiple texts through the content description model to obtain the target text.

[0039] In one possible implementation, the text acquisition module is further used to: input multimedia data and a first instruction into a content description model to obtain a first text output by the content description model; input multimedia data and a second instruction into the content description model to obtain a second text output by the content description model; wherein the multiple instructions include a first instruction and a second instruction, the multiple texts include a first text and a second text, and the second instruction is generated based on the first text.

[0040] In a possible implementation, at least one instruction is a text instruction or a voice instruction.

[0041] In one possible implementation, the processing module is further used to: perform normalization on the first text feature and the first multimedia feature to obtain a second text feature and a second multimedia feature located in the same feature space; obtain text features and multimedia features corresponding to each multimedia data in the other multiple multimedia data in the training data set, wherein the method for obtaining the text feature corresponding to each multimedia data is the same as the method for obtaining the second text feature, and the method for obtaining the multimedia feature corresponding to each multimedia data is the same as the method for obtaining the second multimedia feature; determine the difference between the first text feature and the first multimedia feature based on multiple first relationships and multiple second relationships, wherein the multiple first relationships include a size relationship between the second text feature and the text feature corresponding to each multimedia data, the multiple second relationships include a size relationship between the second multimedia feature and the multimedia feature corresponding to each multimedia data, the multiple first relationships correspond one-to-one to the multiple second relationships, and the number of target relationship pairs is negatively correlated with the difference, and the target relationship pairs include first and second relationships with the same size relationship and corresponding to each other.

[0042] In one possible implementation, the first multimedia feature is an intermediate feature extracted by the first model; the text acquisition module is further used to: obtain a data processing result, which is output after the first model processes the multimedia data; determine a first difference value based on the data processing result and the true label of the multimedia data, and the first difference value is used to represent the difference between the data processing result and the true label of the multimedia data; wherein the loss function value is obtained based on the first difference value and the second difference value, and the second difference value is obtained based on the difference between the first text feature and the first multimedia feature.

[0043] In one possible implementation, the first text feature is extracted by a text feature extraction model; wherein the text feature extraction model is a pre-trained model, or the text feature extraction model is a model trained together with the first model.

[0044] In one possible implementation, the content description model is a pre-trained model. Alternatively, the content description model is a model trained together with the first model, and the loss function used to train the content description model is the same as the loss function used to train the first model.

[0045] In a possible implementation, the number of parameters of the content description model is greater than the number of parameters of the first model.

[0046] In one possible implementation, when the multimedia data is an image, the first model is used to perform any one of the following multimedia data processing tasks: image classification, image segmentation, image recognition, object detection, image generation, and image enhancement;

[0047] Alternatively, when the multimedia data is a video, the first model is used to perform any one of the following multimedia data processing tasks: video classification, video behavior detection, and video generation;

[0048] Alternatively, when the multimedia data is speech, the first model is used to perform any one of the following multimedia data processing tasks: speech recognition and speech synthesis.

[0049] In a third aspect, the present application provides a model training device, which may include a processor coupled to a memory, wherein the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method of the first aspect or any implementation of the first aspect is implemented. For the steps in each possible implementation of the first aspect executed by the processor, please refer to the first aspect for details and will not be repeated here.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method of any implementation of the first aspect.

[0051] A fifth aspect of the present application provides a circuit system, the circuit system including a processing circuit, and the processing circuit is configured to execute a method of any implementation manner of the above-mentioned first aspect.

[0052] In a sixth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of any implementation manner of the first aspect.

[0053] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting a server or a threshold value acquisition device to implement the functions involved in any implementation of the first aspect, for example, sending or processing the data and / or information involved in the above method. In one possible design, the chip system also includes a memory for storing program instructions and data necessary for the server or communication device. The chip system can be composed of a chip or can include a chip and other discrete devices.

[0054] The beneficial effects of the second to seventh aspects mentioned above can be referred to the introduction of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] FIG1 is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0056] FIG2 is a schematic diagram of the structure of another convolutional neural network provided in an embodiment of the present application;

[0057] FIG3 is a schematic diagram of a system architecture 300 provided in an embodiment of the present application;

[0058] FIG4 is a schematic structural diagram of an electronic device 101 provided in an embodiment of the present application;

[0059] FIG5 is a flow chart of a model training method provided in an embodiment of the present application;

[0060] FIG6 is a schematic diagram of a process for training a first model according to an embodiment of the present application;

[0061] FIG7 is a schematic diagram of a process for training a first model according to an embodiment of the present application;

[0062] FIG8 is a schematic diagram of normalizing text features and image features to a spherical space according to an embodiment of the present application;

[0063] FIG9 is a schematic diagram of inputting an image and text into a content description model and outputting a target text from the content description model, provided by an embodiment of the present application;

[0064] FIG10 is a schematic diagram of another embodiment of the present application, in which an image and text are input to a content description model and the content description model outputs a target text;

[0065] FIG11 is a schematic diagram of another embodiment of the present application, in which an image and text are input to a content description model and the content description model outputs a target text;

[0066] FIG12 is a schematic structural diagram of a model training device provided in an embodiment of the present application;

[0067] FIG13 is a schematic diagram of a structure of an execution device provided in an embodiment of the present application;

[0068] FIG14 is a schematic diagram of a structure of a chip provided in an embodiment of the present application;

[0069] FIG15 is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of this application more clear, the embodiments of this application are described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of this application, rather than all embodiments. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0071] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchangeable where appropriate so that the embodiments can be implemented in a sequence other than that illustrated or described in this application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or that are inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units in this application is a logical division. In actual application, there may be other division methods. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. Moreover, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed into multiple circuit units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this application.

[0072] To facilitate understanding, some technical terms involved in the embodiments of this application are first introduced below.

[0073] (1) Neural Network

[0074] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0075] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0076] (2) Deep Neural Networks

[0077] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neural unit in the i-th layer must be connected to any neural unit in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neural unit in the second layer to the 2nd neural unit in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficients from the kth neural unit in the L-1th layer to the jth neural unit in the Lth layer are defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a predictive model with more parameters is more complex and has greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix for all layers of a trained deep neural network (a weight matrix formed by the vectors W of many layers).

[0078] (3) Convolutional Neural Network (CNN)

[0079] A convolutional neural network is a deep neural network with a convolutional architecture. It includes a feature extractor consisting of a convolutional layer and a subsampling layer. The feature extractor can be viewed as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter. A convolutional layer refers to the layer of neural units in a convolutional neural network that performs convolution processing on the input signal (e.g., the first and second convolutional layers in this embodiment). Within a convolutional layer of a convolutional neural network, a neural unit may only be connected to some of the neural units in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neural units. Neural units in the same feature plane share weights, which are referred to as the convolution kernel. Shared weights can be understood as the way image information is extracted, regardless of position. The underlying principle is that the statistical information of one part of the image is the same as that of another part. This means that image information learned in one part can also be applied to another part. Therefore, the same learning can be used to obtain image information at all positions in the image. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0080] Convolution kernels can be initialized as matrices of random size, and during the training process of the convolutional neural network, the convolution kernels can be learned to obtain reasonable weights. In addition, the direct benefit of shared weights is that they reduce the number of connections between the layers of the convolutional neural network, while also reducing the risk of overfitting.

[0081] Specifically, as shown in FIG1 , the convolutional neural network 100 may include an input layer 110 , a convolutional layer / pooling layer 120 , wherein the pooling layer is optional, and a neural network layer 130 .

[0082] Among them, the structure composed of the convolution layer / pooling layer 120 and the neural network layer 130 can be the first convolution layer and the second convolution layer described in this application, the input layer 110 is connected to the convolution layer / pooling layer 120, the convolution layer / pooling layer 120 is connected to the neural network layer 130, the output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform nonlinear processing on the output of the neural network layer 130.

[0083] Convolutional layer / pooling layer 120. Convolutional layer: As shown in Figure 2, convolutional layer / pooling layer 120 may include layers 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of a convolutional layer can serve as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0084] Taking convolution layer 121 as an example, convolution layer 121 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from the input image matrix. A convolution operator is essentially a weight matrix, which is typically predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), to extract specific features from the image. The size of the weight matrix should be related to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used, and multiple weight matrices of the same dimension are applied. The outputs of each weight matrix are stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions also have the same dimensions. The multiple feature maps with the same dimensions extracted are then merged to form the output of the convolution operation.

[0085] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0086] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0087] Pooling layer: Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, that is, as shown in the example 120 in Figure 1, each layer 121-126 can be a convolution layer followed by a pooling layer, or multiple convolution layers can be followed by one or more pooling layers.

[0088] Neural network layer 130: After being processed by the convolution layer / pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. Because as before, the convolution layer / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other relevant information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (131, 132 to 13n as shown in Figure 1) and an output layer 140. The parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, text classification, speech classification, etc.

[0089] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140. The output layer 140 has a loss function value similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (the propagation from 110 to 140 in Figure 2 is forward propagation) is completed, the back propagation (the propagation from 140 to 110 in Figure 2 is back propagation) will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0090] It should be noted that the convolutional neural network 100 shown in Figure 1 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network prediction models. For example, as shown in Figure 2, multiple convolutional layers / pooling layers are run in parallel, and the extracted features are input into the full neural network layer 130 for processing.

[0091] (4) Recurrent Neural Network (RNN)

[0092] A recurrent neural network is a type of recursive neural network that takes sequence data as input, performs recursion in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain-like manner.

[0093] Recurrent neural networks (RNNs) possess memory, parameter sharing, and Turing completeness, giving them advantages in learning nonlinear features of sequences. They have applications in natural language processing (NLP), such as speech recognition, language modeling, and machine translation, and are also used for various time series forecasting tasks. RNNs, constructed by incorporating convolutional neural networks, can handle computer vision problems involving sequential inputs.

[0094] RNNs are designed to process sequential data. In traditional neural network prediction models, each layer is fully connected, from the input layer to the hidden layer to the output layer, and the nodes within each layer are disconnected. However, these standard neural networks are inadequate for many problems. For example, to predict the next word in a sentence, you generally need to use the previous words, because the previous and next words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is dependent on the previous output. Specifically, the network remembers previous information and applies it to the calculation of the current output. In other words, the nodes between hidden layers are no longer disconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the previous hidden layer. In theory, RNNs can process sequence data of any length. However, in practice, to reduce complexity, it is often assumed that the current state is only dependent on a few previous states.

[0095] Specifically, the core of RNN is a directed graph. The chain-connected elements in the expanded directed graph are called recurrent units (RNN cells). Generally, the chain connection formed by recurrent units can be compared to the hidden layer in a feedforward neural network, but in different discussions, the "layer" of RNN may refer to the recurrent unit of a single time step or all recurrent units. Therefore, as a general introduction, the concept of "hidden layer" is avoided here. Given the learning data input in sequence, The expanded length of RNN is The sequence to be processed is usually a time series, and the evolution direction of the sequence is called a time-step.

[0096] (5) Residual Network (ResNet)

[0097] Residual networks are an improved form of convolutional neural networks that are easy to optimize and can improve accuracy by increasing their depth. The residual blocks within residual networks use skip connections, which alleviates the vanishing gradient problem associated with increasing the depth of deep neural networks.

[0098] (6) Attention Network

[0099] Attention networks are network models that utilize the attention mechanism to accelerate model training. Currently, typical attention networks include the Transformer model. Models that utilize the attention mechanism assign different weights to each part of the input sequence, thereby extracting more important features from the input sequence and ultimately achieving more accurate output.

[0100] In deep learning, the attention mechanism can be implemented using a weight vector that describes importance: when predicting or inferring an element, the weight vector determines the correlation between that element and other elements. For example, for a pixel in an image or a word in a sentence, the attention vector can be used to quantitatively estimate the correlation between the target element and other elements, and the weighted sum of the attention vectors can be used as an approximation of the target value.

[0101] The attention mechanism in deep learning simulates the attention mechanism of the human brain. For example, when a person looks at a painting, although their eyes can see the entire painting, when they look closely, their eyes actually focus on only a small part of the pattern. At this time, the human brain focuses primarily on this small part. In other words, when a person observes an image carefully, the human brain's attention is not evenly distributed across the entire image; instead, it is weighted differently. This is the core idea of ​​the attention mechanism.

[0102] Simply put, the human visual processing system tends to selectively focus on certain parts of an image and ignore other irrelevant information, which helps the human brain perceive. Similarly, in some problems involving language, speech, or vision, certain parts of the input may be more relevant than others in deep learning. Therefore, through the attention mechanism in the attention model, the attention model can perform different processing on different parts of the input data, so that the attention model dynamically focuses only on data relevant to the task.

[0103] (7) Pre-trained model

[0104] A pre-trained model is a model that has been pre-trained using training data. Generally speaking, a pre-trained model is trained using a larger training dataset, and the number of parameters in a pre-trained model is usually relatively large.

[0105] (8) Loss function

[0106] During neural network training, because we want the output of the neural network to be as close as possible to the desired predicted value, we compare the current network's prediction with the desired target value and then update the weight vectors of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters of each layer in the neural network.) For example, if the network's prediction is too high, the weight vectors are adjusted to predict a lower value. This adjustment is repeated until the neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (i.e., the loss function value) indicates a greater difference, so neural network training becomes a process of minimizing this loss function value.

[0107] (9) Backpropagation algorithm

[0108] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial prediction model during training, reducing the error loss of the prediction model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial prediction model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal prediction model parameters, such as the weight matrix.

[0109] (10) Gradient descent

[0110] Gradient descent is a first-order optimization algorithm commonly used in machine learning to recursively approximate minimum-deviation prediction models. To find a local minimum of a function using gradient descent, an iterative search must be performed toward a point on the function at a specified step distance in the opposite direction of the gradient (or approximate gradient) corresponding to the current point. Gradient descent is one of the most commonly used methods for solving unconstrained optimization problems, such as predictive model parameters in machine learning algorithms.

[0111] Specifically, when solving for the minimum value of the loss function, we can use the gradient descent method to iterate step by step to obtain the minimized loss function value and prediction model parameter value. Conversely, if we need to solve for the maximum value of the loss function, we need to use the gradient ascent method to iterate.

[0112] For ease of understanding, the following first introduces the system architecture used in the training method of the model provided in the embodiment of the present application.

[0113] Referring to Figure 3 , an embodiment of the present application provides a schematic diagram of a system architecture 300. As shown in Figure 3 , in this system architecture 300, the execution device 210 can be implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, and load balancers. The execution device 210 can be deployed at a single physical site or distributed across multiple physical sites. The execution device 210 can use data in the data storage system 250 or invoke program code in the data storage system 250 to implement the classification model training method provided in the embodiment of the present application, thereby obtaining a model.

[0114] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0115] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0116] In one implementation, the execution device 210 is used to implement the training method of the classification model provided in the embodiment of the present application, and send the obtained model to the local device 301 and the local device 302 through the communication network, so that the local device 301 and the local device 302 can deploy and run the model, thereby implementing the classification method provided in the embodiment of the present application.

[0117] In another implementation, one or more aspects of the execution device 210 can be implemented by each local device. For example, the local device 301 can provide local data or feedback calculation results to the execution device 210, or execute the training method of the model provided in the embodiment of the present application.

[0118] It should be noted that all functions of the execution device 210 can also be implemented by local devices. For example, the local device 301 implements the functions of the execution device 210 and provides services to its own users, or provides services to the users of the local device 302.

[0119] In general, the training method of the model provided in the embodiments of the present application can be applied to electronic devices, such as the execution device 210, local device 301 or local device 302 mentioned above. For example, the electronic device can be, for example, a server, a wireless electronic device in industrial control, a smart phone (mobile phone), a personal computer (PC), a laptop, a tablet computer, an autonomous vehicle, a smart camera, and other devices. For ease of understanding, the method provided in the embodiments of the present application will be described below by taking the application of the method provided in the embodiments of the present application to a server as an example.

[0120] Please refer to Figure 4, which is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of the present application. As shown in Figure 4, electronic device 101 includes a processor 103, which is coupled to a system bus 105. Processor 103 can be one or more processors, each of which can include one or more processor cores. A display adapter (video adapter) 107 can drive a display 109, which is coupled to system bus 105. System bus 105 is coupled to an input / output (I / O) bus via a bus bridge 111. An I / O interface 115 is coupled to the I / O bus. The I / O interface 115 communicates with various I / O devices, such as an input device 117 (e.g., a touch screen), an external memory 121 (e.g., a hard disk, floppy disk, optical disk, or USB flash drive), a multimedia interface, etc., a transceiver 123 (capable of sending and / or receiving radio communication signals), a camera 155 (capable of capturing still and dynamic digital video images), and an external USB port 125. Optionally, the interface connected to the I / O interface 115 may be a USB interface.

[0121] The processor 103 may be any conventional processor, including a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, or a combination thereof. Alternatively, the processor may be a dedicated device such as an ASIC.

[0122] Electronic device 101 can communicate with software deployment server 149 via network interface 129. Exemplarily, network interface 129 is a hardware network interface, such as a network card. Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Alternatively, network 127 can be a wireless network, such as a WiFi network or a cellular network.

[0123] The hard drive interface 131 is coupled to the system bus 105. The hard drive interface is connected to the hard drive 133. The internal memory 135 is coupled to the system bus 105. The data running in the internal memory 135 may include the operating system (OS) 137 of the electronic device 101, the application 143, and the scheduler.

[0124] The operating system consists of a shell 139 and a kernel 141. Shell 139 is an interface between the user and the operating system's kernel. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: it waits for user input, interprets user input to the operating system, and processes various operating system output.

[0125] The kernel 141 consists of the parts of the operating system that manage memory, files, peripherals, and system resources. The kernel 141 directly interacts with the hardware. The operating system kernel typically runs processes and provides inter-process communication, CPU time slice management, interrupts, memory management, and I / O management.

[0126] Please refer to Figure 5, which is a flow chart of a model training method provided in an embodiment of the present application. As shown in Figure 5, the model training method includes the following steps 501-504.

[0127] Step 501: Input multimedia data into a content description model to obtain a target text output by the content description model. The target text is used to describe the content in the multimedia data.

[0128] In this embodiment, the multimedia data is data in the training data set, and is training data used to train the first model. The multimedia data is, for example, images, videos, or voices, and this embodiment does not limit the data type of the multimedia data.

[0129] The content description model can be a model that has been pre-trained (i.e., a pre-trained model); the content description model can also be a model that is trained together with the first model. By inputting multimedia data into the content description model, the content description model can process the multimedia data and output a target text for describing the content in the multimedia data. For example, assuming that the multimedia data is an image taken outdoors, and the content in the image is a puppy running on the grass, then after the multimedia data is input into the content description model, the target text output by the content description model can specifically be "a puppy running on the grass". For another example, assuming that the multimedia data is a video taken outdoors, and the video content is a puppy running on the grass, then the target text output by the content description model can also be "a puppy running on the grass".

[0130] Optionally, the content description model can be a multimodal model capable of processing different types of input data, such as images, videos, text, or voice. Furthermore, the input data of the content description model can indicate the task to be performed by the content description model, so that the content description model can perform the corresponding task processing by recognizing the input data. For example, in this embodiment, a combination of text or voice and multimedia data can be input into the content description model, and the text or voice can be used to instruct the content description model to describe the content in the multimedia data, so that the content description model outputs target text that describes the content in the multimedia data.

[0131] Exemplarily, the content description model can be a generative pre-training self-attention model (Generative Pre-training Transformer, GPT), a guided language image content description model (Bootstrapping Language-Image Pre-training, BLIP2) and a multimodal large model such as mPLUG, that is, a model with a large number of parameters.

[0132] Optionally, to ensure that the target text output by the content description model accurately describes the content of the multimedia data, the content description model has a larger number of parameters than the first model. That is, the content description model is a neural network model with a larger number of parameters. Therefore, the content description model has relatively superior performance and can ensure that the output target text accurately describes the content of the multimedia data.

[0133] Optionally, the content description model can be specifically trained to process multimedia data and output text describing the content within the multimedia data. For example, the input data of the content description model is fixed as images, the output data of the content description model is fixed as text, and the image processing tasks that the content description model can perform are also fixed. That is, the image processing task performed by the content description model is to recognize the input image and output text that can describe the content within the input image.

[0134] Step 502: extract the features of the target text to obtain the first text features.

[0135] In this embodiment, an AI model may be used to extract features of the target text, thereby obtaining the first text features. For example, the features of the target text may be extracted using a text feature extraction model, such as an attention network (e.g., Transformer), a CNN, or other AI model.

[0136] Optionally, the text feature extraction model can be a pre-trained AI model. Specifically, the text feature extraction model can be part of a complete text processing model, and the complete text processing model has been trained, so the weight parameters in the text feature extraction model are also obtained after training.

[0137] The text feature extraction model is responsible for performing feature extraction on the input text and outputting the extracted text features. The other part of the text processing model then continues to process the text based on the text features output by the text feature extraction model and outputs the text processing results. For example, the text processing model can be a text classification model, which extracts and classifies text features based on these features. Thus, the text feature extraction model can be the first half of the model structure in the text classification model, which extracts features from the input text. Furthermore, the text processing model can also perform other text processing tasks, such as text translation or text segmentation, although these are not specifically limited here.

[0138] Optionally, the text feature extraction model can also be a model that is trained together with the first model. Specifically, the text feature extraction model is a pre-built model that can extract features from input text and output the extracted text features. Moreover, the text feature extraction model is a model that has not yet been trained, but is trained together with the first model during the training process of the first model. That is, during the training process of the first model, the first model and the text feature extraction model can be trained simultaneously based on the same loss function to simultaneously update the weight parameters in the first model and the text feature extraction model.

[0139] Step 503: Input the multimedia data into the first model to obtain first multimedia features extracted by the first model.

[0140] In this embodiment, the first model can be a model that performs any multimedia data processing task. Furthermore, when the first model performs the multimedia data processing task, the first model first extracts features from the input multimedia data and then completes the multimedia data processing task based on the extracted multimedia features.

[0141] Exemplarily, when the multimedia data is an image, the first model is used to perform any one of the following multimedia data processing tasks: image classification, image segmentation, image recognition (such as face recognition), target detection, image generation, and image enhancement; or, when the multimedia data is a video, the first model is used to perform any one of the following multimedia data processing tasks: video classification, video behavior detection, and video generation; or, when the multimedia data is speech, the first model is used to perform any one of the following multimedia data processing tasks: speech recognition and speech synthesis. In addition, the first model can also perform other multimedia data processing tasks, which will not be repeated here. Among them, the first model can be, for example, a model such as CNN, Transformer, ResNet, or Multilayer Perceptron (MLP).

[0142] When the multimedia data is an image, the first multimedia feature extracted by the first model is an image feature; when the multimedia data is a video, the first multimedia feature extracted by the first model is a video feature; when the multimedia data is speech, the first multimedia feature extracted by the first model is a speech feature.

[0143] Specifically, the first multimedia feature may be an intermediate feature output by the first model, which is obtained after the feature extraction module in the first model performs feature extraction processing on the multimedia data. Generally speaking, the first model may be composed of multiple neural network blocks in series. A neural network block can describe a single neural network layer, a component composed of multiple neural network layers, or the entire model itself. One advantage of using neural network blocks for abstraction is that some neural network blocks can be combined into larger components. In this embodiment, the first multimedia feature may be a feature between any two adjacent neural network blocks in the first model, that is, the first multimedia feature is a feature output by the previous neural network block and serves as the input of the next neural network block. This embodiment does not limit the network structure used to extract the first multimedia feature.

[0144] Step 504 : updating the first model based on the loss function value to obtain a second model, wherein the loss function value is obtained based on the difference between the first text feature and the first multimedia feature.

[0145] Specifically, during the training process of the first model, a loss function may be constructed first. This loss function is related to the first text feature and the first multimedia feature output by the first model. Based on the difference between the first text feature and the first multimedia feature, a specific loss function value may be calculated. The weight parameters in the first model may then be updated based on the loss function value and a backpropagation algorithm, thereby obtaining a second model. The network structure of the second model is identical to that of the first model, and the weight parameters of the second model are obtained by updating the weight parameters of the first model.

[0146] During the training process of the first model, the training goal of the first model is to reduce the loss function value corresponding to the first model. Since the loss function value is obtained based on the difference between the first text feature and the first multimedia feature, the training goal of the first model is actually to reduce the difference between the first text feature and the first multimedia feature, that is, to make the multimedia features extracted by the first model as close as possible to the text features corresponding to the text output by the content description model. In this way, by constructing a loss function based on the difference between the first text feature and the first multimedia feature, the rich information in the multimedia data identified by the content description model can be used to help the first model realize the recognition of multimedia data, and the content description model with higher performance can be fully utilized to enhance the accuracy of the first model in multimedia data processing tasks.

[0147] It is understandable that the above only introduces the process of performing one round of iterative training on the first model. In actual applications, multiple rounds of iterative training can be performed on the first model based on different training data until the training termination condition is reached (for example, the number of training rounds is greater than the preset number of rounds or the loss function value is less than the preset threshold).

[0148] In general, in this embodiment, multimedia data serving as training data is first input into a content description model. The content description model then outputs text describing the content in the multimedia data, and features are extracted from this text to obtain text features. This multimedia data is then input into the model to be trained, which extracts multimedia features. A loss function is then constructed based on the text features and multimedia features corresponding to the multimedia data, thereby training the model to be trained. Outputting descriptive text specific to the multimedia data from the content description model provides more detailed information about the multimedia data, and constructing a loss function based on text features and multimedia features allows for matching between the text features and multimedia features, helping to enhance the model's understanding of the multimedia data and effectively improving the accuracy of the model's processing of multimedia data.

[0149] Taking the first model as an image classification model (i.e., the first model performs an image classification task) as an example, the traditional image classification model is trained based on a supervised loss function, that is, the supervised loss function is constructed based on the classification results output by the image classification model and the true label (i.e., the category label) of the training data, thereby realizing the training of the image classification model. In other words, in the traditional training method of the image classification model, the image classification model is actually trained from the perspective of the output results of the image classification model, which is equivalent to guiding the final training direction of the image classification model based solely on the category label of the image, and does not provide more detailed help for the training of the image classification model. However, in this embodiment, a content description model is introduced to recognize images, and the content description model outputs text that can describe the specific content in the image, which is equivalent to providing the image with a label containing rich image information. In this way, by matching the text features corresponding to the image and the image features extracted by the image classification model, the image classification model can be guided to learn the detailed information in the cognitive image based on the content description model, providing more detailed help for the image classification model when processing the image classification task, thereby improving the accuracy of the image classification model when performing image classification.

[0150] The above uses the example of the first model performing image classification tasks to introduce the beneficial effects of training the first model based on the above-mentioned model training method. In actual applications, when the first model is used to process other image processing tasks, video processing tasks or voice processing tasks, the model training method provided in this embodiment can also help the first model to better recognize images, videos or voices, thereby improving the accuracy of the first model when performing multimedia data processing tasks. This embodiment will not be repeated here.

[0151] For example, please refer to Figure 6, which is a schematic diagram of a process for training the first model provided in an embodiment of the present application. As shown in Figure 6, the multimedia data used to train the first model can be an image of "a puppy squatting on the lawn". The image is input into the content description model, and the content description model outputs a target text describing the content of the image. The target text can be, for example, "a brown puppy squatting on a green lawn". Secondly, the target text is input into the text feature extraction model to extract the first text feature; the image is input into the first model to obtain the image feature extracted by the first model. In this way, based on the first text feature and the image feature, a loss function can be constructed and the loss function value can be calculated, and the weight parameters of the first model can be updated based on the calculated loss function value.

[0152] In addition, in some embodiments, the total loss function used to train the first model can be composed of two loss functions, one of which is constructed based on the image features extracted by the first model and the text features extracted by the text feature extraction model, and the other loss function is constructed based on the data processing results output by the multimedia data and the true labels of the training data. In this way, by using the total loss function constructed based on the above two loss functions to update the first model, the training of the first model can be guided from the two perspectives of learning to recognize image detail information and learning the final optimization direction, thereby further improving the image processing accuracy of the first model.

[0153] Please refer to Figure 7, which is a schematic diagram of a process for training the first model provided in an embodiment of the present application. As shown in Figure 7, the total loss function used to train the first model is composed of a matching loss function and a supervision loss function, for example, obtained by performing a weighted summation on the matching loss function and the supervision loss function. Among them, the matching loss function can be obtained based on the first text feature extracted by the text feature extraction model and the image feature extracted by the first model (that is, the above-mentioned first multimedia feature). The supervision loss function can be obtained based on the image processing result output by the first model (that is, the above-mentioned data processing result) and the true label corresponding to the image.

[0154] For example, in the above embodiment, after an image is input into the first model, an image processing result can be obtained. This image processing result is output by the first model after processing the image. In other words, the image features are intermediate features output by the first model, while the image processing result is the final result output by the first model.

[0155] Then, based on the image processing result and the true label of the image, a first difference value can be determined, and the first difference value is used to represent the difference between the image processing result and the true label of the image. That is to say, the first difference value is actually the loss function value obtained by substituting the image processing result and the true label of the image into the supervised loss function. Among them, the supervised loss function can be, for example, a cross-entropy loss function, which is not specifically limited in this embodiment. The true label of the image can be a label that is pre-annotated on the image based on the image processing task performed by the first model, which is used to indicate the correct result expected to be output after the first model processes the image. For example, when the first model is used to perform image classification, the true label of the image can be the category label of the image, and the true label of the image shown in Figure 7 is "dog".

[0156] When the first difference value is calculated, a loss function value for updating the first model can be calculated based on the first difference value and the second difference value, for example, by performing a weighted summation on the first difference value and the second difference value. The second difference value is obtained based on the difference between the first text feature and the first multimedia feature, that is, the second difference value is calculated by substituting the first text feature and the first multimedia feature into the above-mentioned matching loss function.

[0157] The following describes a process of determining the second difference value based on the first text feature and the first multimedia feature.

[0158] First, since there are large differences in the distribution of text features and multimedia features (such as image features), the first text feature and the first multimedia feature can be normalized to obtain the second text feature and the second multimedia feature located in the same feature space. Among them, the first text feature and the first multimedia feature are essentially a vector, and the method of normalizing the vector to the spherical space can be specifically to use the vector itself as the numerator and the L2 norm of the vector as the denominator to obtain the normalized vector, thereby realizing the normalization of the vector to the spherical space. For example, as shown in Figure 8, assuming that the multimedia feature is an image feature, the text feature and the image feature can be normalized to the spherical space, thereby realizing the unification of the size and dimension of the two types of features.

[0159] In addition, in addition to normalizing the text features and multimedia features to the spherical space, the text features and multimedia features can also be normalized to other spaces, as long as the normalized text features and the normalized multimedia features are in the same unified space.

[0160] Secondly, for multimedia data, after obtaining the normalized text features (i.e., the second text features) and the normalized multimedia features (the second multimedia features), the text features and multimedia features corresponding to each multimedia data in the other multiple multimedia data in the training data set can be further obtained, wherein the method for obtaining the text features corresponding to each multimedia data is the same as the method for obtaining the second text features, and the method for obtaining the multimedia features corresponding to each multimedia data is the same as the method for obtaining the second multimedia features, and the above-mentioned multimedia data is located in the training data set. That is, for each multimedia data in the training data set, the description text corresponding to each multimedia data is first output through the content description model described in step 501 above, and the text features of the description text corresponding to each multimedia data are output based on the text feature extraction model described in step 502, and the text features output by the text feature extraction model are normalized to obtain the text features corresponding to each multimedia data in the training data set; secondly, feature extraction processing is performed on each multimedia data through the first model, and the extracted multimedia features are normalized to obtain the multimedia features corresponding to each multimedia data. In this way, for each multimedia data in the training data set, the text features and multimedia features corresponding to each multimedia data can be obtained, and these text features and multimedia features have been normalized to the same space.

[0161] Finally, based on the multiple first relationships and the multiple second relationships, the difference between the first text feature and the first multimedia feature is determined. The multiple first relationships include a size relationship between the second text feature and the text feature corresponding to each multimedia data item, i.e., each first relationship indicates a size relationship between the second text feature and the text feature corresponding to a certain multimedia data item, for example, indicating that the second text feature is greater than the text feature corresponding to a certain multimedia data item, or indicating that the second text feature is less than the text feature corresponding to a certain multimedia data item. The multiple second relationships include a size relationship between the second multimedia feature and the multimedia feature corresponding to each multimedia data item, i.e., each second relationship indicates a size relationship between the second multimedia feature and the multimedia feature corresponding to a certain multimedia data item. Since each multimedia data item corresponds to a text feature and a multimedia feature, the multiple first relationships correspond one-to-one with the multiple second relationships. For a set of corresponding first and second relationships, the first relationship indicates a size relationship between the second text feature and the text feature corresponding to a certain multimedia data item, and the second relationship indicates a size relationship between the second multimedia feature and the multimedia feature corresponding to the multimedia data item.

[0162] In addition, the number of target relationship pairs is negatively correlated with the difference between the first text feature and the first multimedia feature, and the target relationship pairs include a first relationship and a second relationship that have the same and corresponding size relationship. For example, assuming that the multimedia data is an image, for image 1 in the training data set, if the size relationship between the second text feature and the text feature corresponding to image 1 (i.e., the first relationship) is such that the former is greater than the latter; and the size relationship between the second multimedia feature and the image feature corresponding to image 1 (i.e., the second relationship) is also such that the former is greater than the latter, then it means that the first relationship and the second relationship related to image 1 indicate the same size relationship.

[0163] That is, the greater the number of target relationship pairs, the closer the relative distance between the first text feature and the text features corresponding to the other images is to the relative distance between the first multimedia feature and the image features corresponding to the other images, and thus the smaller the difference between the first text feature and the first multimedia feature; the fewer the number of target relationship pairs, the further the relative distance between the first text feature and the text features corresponding to the other images is from the relative distance between the first multimedia feature and the image features corresponding to the other images, and thus the larger the difference between the first text feature and the first multimedia feature. A target relationship pair refers to a first relationship and a second relationship having a corresponding relationship indicating the same size relationship.

[0164] Simply put, in this embodiment, the difference between the second text feature and the second multimedia feature is not directly compared. Instead, the second text feature is compared with other text features to determine the size ranking relationship between the second text feature and multiple text features, and the second multimedia feature is compared with other multimedia features to determine the size ranking relationship between the second multimedia feature and multiple multimedia features. If the size ranking relationship between the second text feature and multiple text features is closer to the size ranking relationship between the second multimedia feature and multiple multimedia features, then the difference between the second text feature and the second multimedia feature is smaller; if the size ranking relationship between the second text feature and multiple text features is less close to the size ranking relationship between the second multimedia feature and multiple multimedia features, then the difference between the second text feature and the second multimedia feature is larger.

[0165] In general, the total loss function for training the first model can be expressed by the following formula 1. L0 = W1*L1 + W2*L2 Formula 1

[0166] Where L0 represents the total loss function, L1 represents the supervised loss function, L2 represents the matching loss function, W1 represents the weight of the supervised loss function, and W2 represents the weight of the matching loss function. Specifically, the supervised loss function can be a cross-entropy loss function constructed based on the data processing results output by the first model and the true labels of the multimedia data input to the first model. For details, please refer to the existing definition of the cross-entropy loss function and will not be repeated here.

[0167] The matching loss function can be expressed by the following formula 2.

[0168] Where σ(z)=(az) represents the hinge function, and a is a given constant. If (az)>0, then σ(z)=az, and if (az)≤0, then σ(z)=0. Generally speaking, the value of a can be 1. Therefore, if Then σ(z) is positive if Then σ(z) is 0.

[0169] In addition, sign represents the sign function (+1 or -1), that is, if Greater than 0, then The value of is 1; if Less than 0, then The value of P is -1. img is the image feature after normalization to the spherical space, P text It represents the text features after normalization to the spherical space, and the subscripts i and j represent the sequence numbers of the training samples.

[0170] From Formula 2, we can see that the matching loss function is actually to pair multiple training samples in pairs and determine the loss function value of each two training samples (i.e. ) and.

[0171] Taking images as training samples as an example, for any two training samples, if the size relationship between the image features corresponding to the two training samples is the same as the size relationship between the text features corresponding to the two training samples, then the loss function value corresponding to the two training samples is smaller; if the size relationship between the image features corresponding to the two training samples is different from the size relationship between the text features corresponding to the two training samples, then the loss function value corresponding to the two training samples is larger. In this way, the more training sample pairs there are where the size relationship between the image features and the size relationship between the text features are the same, the smaller the loss function value will be.

[0172] Specifically, during a round of iterative training for the first model, multiple training samples in the training data set are used to train the first model. Therefore, when constructing the matching loss function, the content description model and the text feature extraction model can be first used to process each training sample in the training data set, and ultimately the text features normalized to the spherical space corresponding to each training sample are obtained; and the first model is used to perform feature extraction on each training sample to obtain the image features normalized to the space corresponding to each training sample. In this way, referring to the above formula 2, the text features normalized to the spherical space and the image features normalized to the space corresponding to each training sample are substituted into formula 2 to obtain the value of the matching loss function.

[0173] Simply put, the matching loss function expressed in Formula 2 above can actually be understood as a ranking-based metric function that fully considers the relative relationship between two features rather than the absolute distance. When the first model performs feature extraction on multiple images to obtain multiple image features, and the text feature extraction model performs feature extraction on the descriptive text corresponding to the multiple images to obtain multiple text features, the closer the size ranking relationship between the multiple image features is to the size ranking relationship between the multiple text features, the closer the image features extracted by the first model are to the text features extracted by the text feature extraction model, and therefore the smaller the matching loss function value.

[0174] For example, for images 1-3, the first model extracts image features 1-image features 3, respectively, and the text feature extraction model extracts text features 1-text features 3 corresponding to images 1-image 3, respectively. If the order of image features 1-image 3 is image feature 1 > image feature 2 > image feature 3, and the order of text features 1-text 3 is text feature 1 > text feature 2 > text feature 3, then the image features extracted by the first model are very close to the text features extracted by the text feature extraction model. If the order of image features 1-image 3 is image feature 1 > image feature 2 > image feature 3, and the order of text features 1-text 3 is text feature 1 < text feature 2 < text feature 3, then the image features extracted by the first model differ significantly from the text features extracted by the text feature extraction model.

[0175] In the above embodiment, when the content description model is a multimodal model, there are multiple ways to obtain the target text based on the content description model. The following will detail the specific implementation of obtaining the target text for describing multimedia data based on the content description model.

[0176] Implementation method 1 is to input multimedia data and an instruction into a content description model to obtain a target text output by the content description model, wherein the instruction input into the content description model is used to instruct the content description model to describe the content in the multimedia data.

[0177] For example, the instructions input into the content description model can be text instructions or voice instructions, i.e., text or voice instructions are input into the content description model as instructions. Furthermore, the instructions input into the content description model can be determined based on the specific type of multimedia data processing task of the first model, and this embodiment does not specifically limit this.

[0178] Since the content description model is a multimodal model, it can receive data in multiple different modalities as input data and process data in different modalities. Therefore, the content description model can recognize the content indicated by the text instructions or voice instructions, and then recognize the multimedia data based on the content indicated by the text instructions or voice instructions and output the target text used to describe the content in the multimedia data.

[0179] Simply put, this embodiment actually uses the content description model as a visual question-answering system. By inputting multimedia data and custom questions (i.e., text instructions or voice instructions) into the content description model, the content description model outputs a description of the multimedia data.

[0180] Exemplarily, please refer to Figure 9, which is a schematic diagram of an embodiment of the present application, in which an image and text are input into a content description model and a target text is output by the content description model. As shown in Figure 9, an image and a text instruction are input into the content description model at the same time. The image is a photo taken in a traffic scene, and the text instruction is specifically "what is contained in the image", that is, the text instruction requests the content description model to describe what is contained in the image. After the image and text instruction are input into the content description model, the target text output by the content description model is "car, road, zebra crossing, traffic light". In other words, the content description model recognizes the image according to the instructions of the text instruction, and lists all the objects contained in the image in the form of target text.

[0181] It should be understood that FIG9 illustrates a text instruction indicating only one question, meaning that the content description model only needs to answer the one question indicated by the text instruction to complete the image description. In some embodiments, to enable the content description model to output target text containing more image details, that is, to output target text that more fully describes the image, the instruction input to the content description model may include multiple questions simultaneously, thereby instructing the content description model to simultaneously describe the content of the image from the perspectives defined by the multiple questions.

[0182] For example, please refer to Figure 10, which is a schematic diagram of another embodiment of the present application, providing an example of inputting an image and text into a content description model and outputting target text from the content description model. As shown in Figure 10, an image and a text instruction are simultaneously input into the content description model. The image is a photograph taken in a traffic scene.

[0183] Among them, the text instruction is specifically "Please describe the content in the picture from the following perspectives: 1. What objects are included in the picture? 2. What color are the objects in the picture? 3. What actions are the objects in the picture performing?". In other words, the text instruction actually includes multiple questions, and based on the multiple questions, the content description model is requested to describe the content contained in the multimedia data from different perspectives, thereby enabling the content description model to output a target text including more detailed content (i.e., a text that can describe more detailed information in the image). Specifically, after the image and text instruction are input into the content description model, the target text output by the content description model is "On the gray road, a blue car stops in front of a white zebra crossing waiting for the red traffic light to turn green." Compared with the example shown in Figure 9, it can be seen that in Figure 10, multiple questions are defined simultaneously in the text instruction, which can effectively instruct the content description model to describe the content in the image from different perspectives, thereby obtaining a target text containing more detailed image information.

[0184] Implementation method 2 is to input multimedia data and multiple instructions into a content description model to obtain multiple texts output by the content description model, and perform fusion processing on the multiple texts through the content description model to obtain a target text.

[0185] Each of the multiple instructions input to the content description model is used to instruct the content description model to describe the content of the multimedia data, and different instructions instruct the content description model to describe different contents. When multiple instructions are input to the content description model, the content description model processes each input instruction and outputs a corresponding text for each instruction. In this way, for the multiple input instructions, the content description model can output multiple texts corresponding to the multiple instructions.

[0186] Furthermore, because different instructions instruct the content description model to describe different contents in the image (i.e., instruct the content description model), different texts in the multiple texts output by the content description model also describe different contents in the image. This allows the content description model to fuse the multiple texts, thereby outputting target text that describes the image content from different perspectives, i.e., containing more detailed image information.

[0187] Among them, multiple instructions can be input into the content description model simultaneously with the multimedia data, or they can be combined with the multimedia data separately and then input into the content description model in sequence. This embodiment does not specifically limit the way in which multiple instructions are input into the content description model.

[0188] For example, assuming that the multiple instructions include a first instruction and a second instruction, and the multiple texts corresponding to the multiple instructions include the first text and the second text. Then, the specific method for inputting the multiple instructions and multimedia data into the content description model and obtaining the multiple texts output by the content description model can be: first, input the multimedia data and the first instruction into the content description model to obtain the first text output by the content description model; then, input the multimedia data and the second instruction into the content description model to obtain the second text output by the content description model. In other words, each of the multiple instructions needs to be combined with the multimedia data before being input into the content description model, and the multiple instructions are input into the content description model sequentially.

[0189] Furthermore, the second instruction can be generated based on the first text. That is, after a combination of an instruction and multimedia data is input into the content description model and the text output by the content description model is obtained, the next instruction to be input into the content description model can be generated based on the text output by the content description model, so that the content description model can more accurately understand the content to be described.

[0190] For example, please refer to Figure 11, which is a schematic diagram of another embodiment of the present application for inputting an image and text into a content description model and outputting a target text from the content description model. As shown in (a) in Figure 11, first, an image and a text instruction 1 are simultaneously input into the content description model. The image is a photo taken of a traffic scene, and the text instruction 1 is specifically "What is contained in the image?" Based on the input image and text instruction 1, the content description model can output a text 1 for describing the image based on the instructions of the text instruction 1. The text 1 is specifically "car, road, zebra crossing, traffic light".

[0191] As shown in Figure 11(b), based on the text 1 output by the content description model, a corresponding text instruction 2 can be generated. This text instruction 2 specifically reads, "What are the colors of the car, road, zebra crossing, and traffic light in the image?" In other words, text instruction 2 actually instructs the content description model to describe the colors of the objects included in text 1. Furthermore, text instruction 2 and the image are combined and input into the content description model, which then outputs text 2. Specifically, text 2 reads, "The car is blue, the road is gray, the zebra crossing is white, and the traffic light is red."

[0192] As shown in Figure 11(c), based on the text 1 output by the content description model, a corresponding text instruction 3 can be generated. This text instruction 3 specifically reads, "What is the interactive relationship between the car, road, zebra crossing, and traffic light in the image?" Furthermore, the text instruction 3 and the image are combined and input into the content description model, which then outputs text 3. Specifically, text 3 reads, "The car stopped at the zebra crossing on the road, waiting for the traffic light to turn green."

[0193] As shown in (d) of Figure 11, after obtaining text 1-text 3 output by the content description model based on text instructions 1-text instructions 3, text instruction 4 can be generated based on text 1-text 3 and input into the content description model, thereby requesting the content description model to perform fusion processing on text 1-text 3. Specifically, the content of text instruction 4 is "Please merge the following sentences: 1, car, road, zebra crossing, traffic light; 2, the car is blue, the road is gray, the zebra crossing is white, and the traffic light is red; 3, the car stops in front of the zebra crossing on the road and waits for the traffic light to turn green." Finally, the text 4 output by the content description model is "On the gray road, the blue car stops in front of the white zebra crossing and waits for the red traffic light to turn green."

[0194] In general, this embodiment uses a content description model that can output text describing the image content as a teaching assistant model to assist in the training of the image processing model (i.e., the first model mentioned above), which helps the image processing model learn how to recognize more image detail information during the training process, thereby improving the accuracy of the image processing model.

[0195] For example, as shown in Table 1, Table 1 is a comparison of the classification accuracy between image classification models obtained after training image classification models of the same structure using the relevant technology and the model training method provided by this embodiment.

[0196] Table 1

[0197] As can be seen from Table 1, compared with the related art that only uses the supervised loss function to train the image classification model, the image classification model trained based on the method of this embodiment has higher accuracy and higher classification precision.

[0198] The above describes in detail the method provided by the embodiment of the present application. Next, the device provided by the embodiment of the present application for executing the above method will be introduced.

[0199] Please refer to Figure 12, which is a schematic diagram of the structure of a training device for a model provided in an embodiment of the present application. As shown in Figure 12, the training device for the model includes: a text acquisition module 1201, which is used to input multimedia data into a content description model to obtain a target text output by the content description model, wherein the target text is used to describe the content in the multimedia data; a processing module 1202, which is used to extract features of the target text to obtain a first text feature; the processing module 1202 is also used to input multimedia data into the first model to obtain a first multimedia feature extracted by the first model; the processing module 1202 is also used to update the first model based on a loss function value to obtain a second model, wherein the loss function value is obtained based on the difference between the first text feature and the first multimedia feature.

[0200] In one possible implementation, the content description model is a multimodal model that can process different types of input data; the text acquisition module 1201 is also used to input multimedia data and at least one instruction into the content description model to obtain the target text output by the content description model, wherein the at least one instruction is used to instruct the content description model to describe the content in the multimedia data.

[0201] In one possible implementation, when at least one instruction is multiple instructions, the text acquisition module 1201 is further used to: input multimedia data and multiple instructions into a content description model to obtain multiple texts output by the content description model, wherein each of the multiple instructions is used to instruct the content description model to describe the content in the multimedia data, and different instructions instruct the content description model to describe different contents, and the multiple instructions correspond to multiple texts; and perform fusion processing on the multiple texts through the content description model to obtain the target text.

[0202] In one possible implementation, the text acquisition module 1201 is further used to: input multimedia data and a first instruction into a content description model to obtain a first text output by the content description model; input multimedia data and a second instruction into the content description model to obtain a second text output by the content description model; wherein the multiple instructions include a first instruction and a second instruction, the multiple texts include a first text and a second text, and the second instruction is generated based on the first text.

[0203] In a possible implementation, at least one instruction is a text instruction or a voice instruction.

[0204] In one possible implementation, the processing module 1202 is further used to: perform normalization on the first text feature and the first multimedia feature to obtain a second text feature and a second multimedia feature located in the same feature space; obtain text features and multimedia features corresponding to each multimedia data in the other multiple multimedia data in the training data set, wherein the method for obtaining the text feature corresponding to each multimedia data is the same as the method for obtaining the second text feature, and the method for obtaining the multimedia feature corresponding to each multimedia data is the same as the method for obtaining the second multimedia feature; determine the difference between the first text feature and the first multimedia feature based on multiple first relationships and multiple second relationships, wherein the multiple first relationships include a size relationship between the second text feature and the text feature corresponding to each multimedia data, the multiple second relationships include a size relationship between the second multimedia feature and the multimedia feature corresponding to each multimedia data, the multiple first relationships correspond one-to-one to the multiple second relationships, and the number of target relationship pairs is negatively correlated with the difference, and the target relationship pairs include first relationships and second relationships with the same size relationship and corresponding to each other.

[0205] In one possible implementation, the first multimedia feature is an intermediate feature extracted by the first model; the text acquisition module 1201 is also used to: obtain a data processing result, which is output after the first model processes the multimedia data; determine a first difference value based on the data processing result and the true label of the multimedia data, and the first difference value is used to represent the difference between the data processing result and the true label of the multimedia data; wherein the loss function value is obtained based on the first difference value and the second difference value, and the second difference value is obtained based on the difference between the first text feature and the first multimedia feature.

[0206] In one possible implementation, the first text feature is extracted by a text feature extraction model; wherein the text feature extraction model is a pre-trained model, or the text feature extraction model is a model trained together with the first model.

[0207] In one possible implementation, the content description model is a pre-trained model. Alternatively, the content description model is a model trained together with the first model, and the loss function used to train the content description model is the same as the loss function used to train the first model.

[0208] In a possible implementation, the number of parameters of the content description model is greater than the number of parameters of the first model.

[0209] In one possible implementation, when the multimedia data is an image, the first model is used to perform any one of the following multimedia data processing tasks: image classification, image segmentation, image recognition, object detection, image generation, and image enhancement;

[0210] Alternatively, when the multimedia data is a video, the first model is used to perform any one of the following multimedia data processing tasks: video classification, video behavior detection, and video generation;

[0211] Alternatively, when the multimedia data is speech, the first model is used to perform any one of the following multimedia data processing tasks: speech recognition and speech synthesis.

[0212] Please refer to Figure 13, which is a structural diagram of an execution device provided in an embodiment of the present application. The execution device 1300 can be specifically manifested as a mobile phone, a tablet, a laptop computer, a smart wearable device, a server, etc., which is not limited here. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303 and a memory 1304 (wherein the number of processors 1303 in the execution device 1300 can be one or more, and Figure 13 takes one processor as an example), wherein the processor 1303 may include an application processor 13031 and a communication processor 13032. In some embodiments of the present application, the receiver 1301, the transmitter 1302, the processor 1303 and the memory 1304 may be connected via a bus or other means.

[0213] Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303. A portion of memory 1304 may also include non-volatile random access memory (NVRAM). Memory 1304 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0214] Processor 1303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0215] The method disclosed in the above embodiment of the present application can be applied to the processor 1303, or implemented by the processor 1303. The processor 1303 can be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 1303 or an instruction in the form of software. The above-mentioned processor 1303 can be a general-purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0216] The processor 1303 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1304, and the processor 1303 reads the information in the memory 1304 and completes the steps of the above method in combination with its hardware.

[0217] Receiver 1301 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1302 can be used to output digital or character information through the first interface. Transmitter 1302 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1302 can also include a display device such as a display screen.

[0218] The electronic device provided in the embodiment of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the method for determining the model structure described in the above embodiment, or so that the chip in the training device executes the method for determining the model structure described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0219] Specifically, see Figure 14 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 1400. NPU 1400 is mounted on the host CPU (host CPU) as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1403, which is controlled by controller 1404 to extract matrix data from memory and perform multiplication operations.

[0220] In some implementations, the arithmetic circuit 1403 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1403 is a two-dimensional systolic array. The arithmetic circuit 1403 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1403 is a general-purpose matrix processor.

[0221] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1408.

[0222] Unified memory 1406 is used to store input and output data. Weight data is directly transferred to weight memory 1402 through the Direct Memory Access Controller (DMAC) 1405. Input data is also transferred to unified memory 1406 through the DMAC.

[0223] BIU stands for Bus Interface Unit, i.e., bus interface unit 1410 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1409 .

[0224] The bus interface unit 1410 (BIU) is used for the instruction fetch memory 1409 to obtain instructions from the external memory, and is also used for the storage unit access controller 1405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0225] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1406 or transfer weight data to the weight memory 1402 or transfer input data to the input memory 1401.

[0226] The vector calculation unit 1407 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1403, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0227] In some implementations, the vector calculation unit 1407 can store the processed output vector to the unified memory 1406. For example, the vector calculation unit 1407 can apply a linear function or a nonlinear function to the output of the operation circuit 1403, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1403, for example, for use in subsequent layers in a neural network.

[0228] An instruction fetch buffer 1409 connected to the controller 1404 is used to store instructions used by the controller 1404;

[0229] Unified memory 1406, input memory 1401, weight memory 1402, and instruction fetch memory 1409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0230] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0231] Please refer to Figure 15, which is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. The present application also provides a computer-readable storage medium. In some embodiments, the method disclosed in Figure 5 above can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or products.

[0232] 15 schematically illustrates a conceptual partial view of an example computer-readable storage medium including a computer program for executing a computer process on a computing device, arranged in accordance with at least some embodiments presented herein.

[0233] In one embodiment, computer readable storage medium 1500 is provided using signal bearing medium 1501. Signal bearing medium 1501 may include one or more program instructions 1502 that, when executed by one or more processors, may provide the functionality or portions of the functionality described above with respect to FIG.

[0234] In some examples, the signal bearing medium 1501 may include a computer readable medium 1503 such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, a ROM or RAM, and the like.

[0235] In some embodiments, the signal-bearing medium 1501 may include a computer-recordable medium 1504, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, or the like. In some embodiments, the signal-bearing medium 1501 may include a communication medium 1505, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, or the like). Thus, for example, the signal-bearing medium 1501 may be communicated via a wireless form of the communication medium 1505 (e.g., a wireless communication medium conforming to the IEEE 802.11 standard or other transmission protocol).

[0236] The one or more program instructions 1502 may be, for example, computer-executable instructions or logic-implemented instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to the program instructions 1502 communicated to the computing device via one or more of computer-readable media 1503, computer-recordable media 1504, and / or communication media 1505.

[0237] It should also be noted that the device embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0238] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods of each embodiment of the present application.

[0239] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0240] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, training device or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center that includes one or more available media integrations. Available media can be magnetic media, (such as floppy disks, hard disks, tapes), optical media (such as DVDs), or semiconductor media (such as solid-state drives (SSDs)).

Claims

1. A model training method, characterized in that: include: Inputting multimedia data into a content description model to obtain a target text output by the content description model, wherein the target text is used to describe the content in the multimedia data; Extracting the feature of the target text to obtain a first text feature; Inputting the multimedia data into a first model to obtain first multimedia features extracted by the first model, wherein the first model is used to perform a multimedia data processing task; Based on the loss function value, the first model is updated to obtain a second model, wherein the loss function value is obtained based on the difference between the first text feature and the first multimedia feature.

2. The method according to claim 1, characterized in that The content description model is a multimodal model, and the multimodal model can process different types of input data; The step of inputting multimedia data into a content description model to obtain a target text output by the content description model includes: The multimedia data and at least one instruction are input into the content description model to obtain the target text output by the content description model, wherein the at least one instruction is used to instruct the content description model to describe the content in the multimedia data.

3. The method according to claim 2, characterized in that In the case where the at least one instruction is a plurality of instructions, the step of inputting the multimedia data and the at least one instruction into the content description model to obtain the target text output by the content description model includes: Inputting the multimedia data and the plurality of instructions into the content description model to obtain a plurality of texts output by the content description model, wherein each of the plurality of instructions is used to instruct the content description model to describe content in the multimedia data, and different instructions instruct the content description model to describe different contents, and the plurality of instructions correspond to the plurality of texts; The target text is obtained by performing fusion processing on the multiple texts through the content description model.

4. The method according to claim 3, characterized in that The step of inputting the multimedia data and the plurality of instructions into the content description model to obtain the plurality of texts output by the content description model comprises: Inputting the multimedia data and the first instruction into the content description model to obtain a first text output by the content description model; Inputting the multimedia data and the second instruction into the content description model to obtain a second text output by the content description model; The multiple instructions include the first instruction and the second instruction, the multiple texts include the first text and the second text, and the second instruction is generated based on the first text.

5. The method according to any one of claims 2 to 4, characterized in that: The at least one instruction is a text instruction or a voice instruction.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Normalizing the first text feature and the first multimedia feature to obtain a second text feature and a second multimedia feature located in the same feature space; Acquire text features and multimedia features corresponding to each multimedia data in the other multiple multimedia data in the training data set, wherein the method for acquiring the text features corresponding to each multimedia data is the same as the method for acquiring the second text features, and the method for acquiring the multimedia features corresponding to each multimedia data is the same as the method for acquiring the second multimedia features; Based on multiple first relationships and multiple second relationships, the difference between the first text feature and the first multimedia feature is determined, wherein the multiple first relationships include the size relationship between the second text feature and the text feature corresponding to each multimedia data, the multiple second relationships include the size relationship between the second multimedia feature and the multimedia feature corresponding to each multimedia data, the multiple first relationships correspond to the multiple second relationships one-to-one, and the number of target relationship pairs is negatively correlated with the difference, and the target relationship pairs include the first relationship and the second relationship that have the same size relationship and correspond to each other.

7. The method according to any one of claims 1 to 6, characterized in that: The first multimedia feature is an intermediate feature extracted by the first model; The method further comprises: Acquire a data processing result, where the data processing result is output after the first model processes the multimedia data; Determine a first difference value based on the data processing result and the real label of the multimedia data, where the first difference value is used to represent the difference between the data processing result and the real label of the multimedia data; The loss function value is obtained based on the first difference value and the second difference value, and the second difference value is obtained based on the difference between the first text feature and the first multimedia feature.

8. The method according to any one of claims 1 to 7, characterized in that: The first text feature is extracted by a text feature extraction model; The text feature extraction model is a pre-trained model, or the text feature extraction model is a model trained together with the first model.

9. The method according to any one of claims 1 to 8, characterized in that: The content description model is a pre-trained model, or the content description model is a model trained together with the first model.

10. The method according to any one of claims 1 to 9, characterized in that: The number of parameters of the content description model is greater than the number of parameters of the first model.

11. The method according to any one of claims 1 to 10, characterized in that: The multimedia data is images, videos or voices.

12. The method according to any one of claims 1 to 11, characterized in that: When the multimedia data is an image, the first model is used to perform any one of the following multimedia data processing tasks: image classification, image segmentation, image recognition, object detection, image generation, and image enhancement; Alternatively, when the multimedia data is a video, the first model is used to perform any one of the following multimedia data processing tasks: video classification, video behavior detection, and video generation; Alternatively, when the multimedia data is speech, the first model is used to perform any one of the following multimedia data processing tasks: speech recognition and speech synthesis.

13. A model training device, characterized in that: include: A text acquisition module, used for inputting multimedia data into a content description model to obtain a target text output by the content description model, wherein the target text is used for describing the content in the multimedia data; A processing module, used for extracting the feature of the target text to obtain a first text feature; The processing module is further used to input the multimedia data into a first model to obtain first multimedia features extracted by the first model; The processing module is further used to update the first model based on a loss function value to obtain a second model, wherein the loss function value is obtained based on the difference between the first text feature and the first multimedia feature.

14. The device according to claim 13, characterized in that The content description model is a multimodal model, and the multimodal model can process different types of input data; The text acquisition module is further used to input the multimedia data and at least one instruction into the content description model to obtain the target text output by the content description model, wherein the at least one instruction is used to instruct the content description model to describe the content in the multimedia data.

15. The device according to claim 14, characterized in that In the case where the at least one instruction is a plurality of instructions, the text acquisition module is further used to: Inputting the multimedia data and the plurality of instructions into the content description model to obtain a plurality of texts output by the content description model, wherein each of the plurality of instructions is used to instruct the content description model to describe content in the multimedia data, and different instructions instruct the content description model to describe different contents, and the plurality of instructions correspond to the plurality of texts; The target text is obtained by performing fusion processing on the multiple texts through the content description model.

16. The device according to claim 15, characterized in that The text acquisition module is also used for: Inputting the multimedia data and the first instruction into the content description model to obtain a first text output by the content description model; Inputting the multimedia data and the second instruction into the content description model to obtain a second text output by the content description model; The multiple instructions include the first instruction and the second instruction, the multiple texts include the first text and the second text, and the second instruction is generated based on the first text.

17. The device according to any one of claims 14 to 16, characterized in that: The at least one instruction is a text instruction or a voice instruction.

18. The device according to any one of claims 13 to 17, characterized in that: The processing module is further used for: Normalizing the first text feature and the first multimedia feature to obtain a second text feature and a second multimedia feature located in the same feature space; Acquire text features and multimedia features corresponding to each multimedia data in the other multiple multimedia data in the training data set, wherein the method for acquiring the text features corresponding to each multimedia data is the same as the method for acquiring the second text features, and the method for acquiring the multimedia features corresponding to each multimedia data is the same as the method for acquiring the second multimedia features; Based on multiple first relationships and multiple second relationships, the difference between the first text feature and the first multimedia feature is determined, wherein the multiple first relationships include the size relationship between the second text feature and the text feature corresponding to each multimedia data, the multiple second relationships include the size relationship between the second multimedia feature and the multimedia feature corresponding to each multimedia data, the multiple first relationships correspond to the multiple second relationships one-to-one, and the number of target relationship pairs is negatively correlated with the difference, and the target relationship pairs include the first relationship and the second relationship that have the same size relationship and correspond to each other.

19. The device according to any one of claims 13 to 18, characterized in that: The first multimedia feature is an intermediate feature extracted by the first model; the text acquisition module is further used to: Acquire a data processing result, where the data processing result is output after the first model processes the multimedia data; Determine a first difference value based on the data processing result and the real label of the multimedia data, where the first difference value is used to represent the difference between the data processing result and the real label of the multimedia data; The loss function value is obtained based on the first difference value and the second difference value, and the second difference value is obtained based on the difference between the first text feature and the first multimedia feature.

20. A model training device, characterized in that: The device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the device executes the method according to any one of claims 1 to 12.

21. A computer storage medium, characterized in that The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 12.

22. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 12.

Citation Information

Patent Citations

  • Method and device for generating description information of multimedia data, equipment and medium

    CN111723937A

  • Multimedia file classification, information processing and model training method, system and equipment

    CN112231275A

  • Model training method and related equipment

    CN114187486A

  • Model training method, article identification method and device, electronic equipment and medium

    CN115310547A

  • Feature extraction model training method and device, equipment and storage medium

    CN116089651A

Cited By

  • Multi-type robot collaborative operation method based on dynamic environment perception modeling

    CN119376326A

  • Video text method and device based on time sequence motion perception and electronic equipment

    CN120853083A