Model training method, device, electronic device and storage medium

Abstract

The present application discloses a model training method, device, electronic device and storage medium, and relates to the field of machine learning technology. The method includes: obtaining training text and training images, wherein the training text is used to describe the content in the training image, parsing the training text, obtaining feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information, inputting the training text and training image into the backbone network of the initial model, and inputting the feature information into the detection head of the initial model, training the initial model, and obtaining a visual semantic understanding model. The embodiment of the present application parses the feature information of the training text as the input of the detection head of the model to supervise key information during the model training process, which can improve the controllability and accuracy of the model training.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and more specifically, to a model training method, device, electronic device and storage medium. Background Art

[0002] Visual semantic understanding is a general term for a series of computer vision tasks, which requires understanding the information contained in computer vision content and outputting it as required. Among them, tasks such as image multi-label classification, interactive character recognition, image and text retrieval, and picture description all belong to this category. However, the current recognition accuracy of visual semantic understanding is poor. Summary of the invention

[0003] In view of the above problems, the present application proposes a model training method, device, electronic device and storage medium to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a method for training a model, the method comprising:

[0005] Acquire training text and training images, wherein the training text is used to describe the content in the training image; parse the training text to acquire feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information; input the training text and the training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model, train the initial model, and obtain a visual semantic understanding model.

[0006] In the second aspect, an embodiment of the present application provides a model training device, which includes: an information acquisition module, used to acquire training text and training images, wherein the training text is used to describe the content in the training image; an information parsing module, used to parse the training text and acquire feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information; a model training module, used to input the training text and the training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model, to train the initial model and obtain a visual semantic understanding model.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0009] The model training method, device, electronic device and storage medium provided in the embodiments of the present application obtain training text and training images, wherein the training text is used to describe the content in the training image, parse the training text, obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information, input the training text and training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model, train the initial model, and obtain a visual semantic understanding model, so as to realize supervision of key information during the model training process by parsing the feature information of the training text as the input of the detection head of the model, thereby improving the controllability and accuracy of the model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 A schematic diagram of a flow chart of a model training method provided in an embodiment of the present application is shown;

[0012] Figure 2 A schematic diagram of a backbone network of a model provided in an embodiment of the present application is shown;

[0013] Figure 3 A schematic diagram of a flow chart of a model training method provided in an embodiment of the present application is shown;

[0014] Figure 4 A schematic diagram of a flow chart of a model training method provided in an embodiment of the present application is shown;

[0015] Figure 5 A schematic diagram of a flow chart of a model training method provided in an embodiment of the present application is shown;

[0016] Figure 6 A schematic diagram showing the framework of the visual semantic understanding model provided by an embodiment of the present application;

[0017] Figure 7 A module block diagram of a model training device provided in an embodiment of the present application is shown;

[0018] Figure 8A block diagram of an electronic device according to an embodiment of the present application for executing a training method for a model according to an embodiment of the present application is shown;

[0019] Fig. 9 A storage unit of an embodiment of the present application is shown for storing or carrying program code for implementing a training method for a model according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0021] Visual semantic understanding is a general term for a series of computer vision tasks, which requires computers to understand the information contained in visual content (objects, scenes, attributes, relationships, etc.) and output them as required. Tasks such as multi-label image classification, interactive character recognition, image and text retrieval, and picture description all fall into this category. At present, there are two problems with visual semantic understanding technology:

[0022] First, the efficiency issue. Based on the correspondence between models and tasks, current technologies can be divided into two categories:

[0023] a) A model can only solve one task. To solve multiple tasks, multiple models need to be trained separately.

[0024] b) A model can solve multiple downstream tasks in a "pre-training + fine-tuning" manner. In essence, each task still requires training a different set of model parameters.

[0025] Therefore, current technology requires a lot of computing power and labeled data of different task types to support it.

[0026] Second, the problem of controllability. Current technology has uncontrollable problems:

[0027] a) The results are uncontrollable. Coarse-grained text and images are often used for training, but there is no alignment between the parts of the text and the image, which makes it difficult to ensure the accuracy of key information.

[0028] b) The process is uncontrollable. The training and reasoning of the model are end-to-end. There is no information that can be used to diagnose the performance of the model. If the results are irrational, it is difficult to locate the problem and it is difficult to perform effective iteration.

[0029] In response to the above problems, the inventors have found through detailed research and proposed the training method, device, electronic device and storage medium of the model provided in the embodiments of the present application. By parsing the feature information of the training text as the input of the detection head of the model, the key information can be supervised during the model training process, which can improve the controllability and accuracy of the model training. Among them, the specific model training method is described in detail in the subsequent embodiments.

[0030] The following is an explanation of the words that can be used in the embodiments of the present application.

[0031] Tags: words that represent objects, scenes, actions, etc., such as "person", "sky", "eating", etc.

[0032] Attributes: words that describe the state of a label, such as color, quantity, material, etc., such as "red", "several", and "wooden";

[0033] Relation, which refers to verbs, locative words, and other words that indicate a subordinate relationship, such as "ride", "on top of", and "belong to";

[0034] Image Tagging refers to identifying one or more labels in an image. The labels here are broad, including the previous labels, attributes, and partial relationships.

[0035] Human Object Interaction Recognition (HOI Recognition) refers to the recognition of one or more action phrases in an image, which can be understood as a combination of relation+tag, such as "ridehorse", "eat pizza", etc.

[0036] Image-Text Retrieval means that given an image, the relevant text is returned from a collection of texts, sorted from high to low in terms of relevance; or, given a text, the relevant images are returned from a collection of images, sorted from high to low in terms of relevance;

[0037] Image Caption means generating a sentence describing the content of a given image.

[0038] See also Figure 1 , Figure 1The flowchart of the training method of the model provided by an embodiment of the present application is shown. The method is used to parse the feature information of the training text as the input of the detection head of the model to realize the supervision of key information during the model training process, which can improve the controllability and accuracy of the model training. In a specific embodiment, the training method of the model is applied to Figure 7 The training device 200 of the model shown and the electronic device 100 ( Figure 8 ). It is understandable that the electronic device may include a smart phone, a tablet computer, a wearable electronic device, a server, etc., which is not limited here. Figure 1 The process shown is described in detail, and the training method of the model can specifically include the following steps:

[0039] Step S110: Acquire training text and training image, wherein the training text is used to describe the content in the training image.

[0040] In this embodiment, a training text and a training image may be obtained, wherein the training text is used to describe the content in the training image. For example, the training text may include "a man is eating pizza", and the training image may include a picture of a full banquet, and the picture of the full banquet includes a man eating pizza.

[0041] In some implementations, there are multiple training texts, multiple training images, and there is a correspondence between the multiple training texts and the multiple training images, wherein the correspondence may include one training text corresponding to one training image, one training text corresponding to multiple training images, or multiple training texts corresponding to one training image, etc., which are not limited here. As a way, the number of training texts and the number of training images can be set according to training requirements, which are not limited here.

[0042] In some embodiments, the training text and training images can be obtained from the server via a network, for example, the training text and training images can be obtained from the server via a wireless network, the training text and training images can be obtained from the server via a wired network, the training text and training images can be obtained from the server via a data network, etc.; the training text and training images can be obtained from other devices via a network, for example, the training text and training images can be obtained from other devices via a wireless network, the training text and training images can be obtained from other devices via a wired network, the training text and training images can be obtained from other devices via a data network, etc.; the training text and training images can also be obtained directly from the local electronic device, etc., which is not limited here.

[0043] As a method, for non-private text and non-private images, such as public text and images, the electronic device can automatically collect text from the network as training text and collect images as training images, and can also automatically obtain text from the electronic device as training text and obtain images as training images. Alternatively, the electronic device can collect text from the network as training text and collect images as training images according to the task instructions issued by the user, and can also collect text from the electronic device as training text and collect images as training images according to the task instructions issued by the user.

[0044] As another way, for private text and private images, such as encrypted text and images, the electronic device can automatically collect text from the network as training text and collect images as training images based on authorization, and can also automatically obtain text from the local electronic device as training text and obtain images as training images. Alternatively, based on authorization, the electronic device can collect text from the network as training text and collect images as training images according to the task instructions issued by the user, and can also obtain text from the local electronic device as training text and obtain images as training images according to the task instructions issued by the user.

[0045] Step S120: parsing the training text to obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information.

[0046] In this embodiment, when a training text is obtained, the training text can be parsed to obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information, and relationship information. As an implementable method, when a training text is obtained, the cost of parsing the training text can be used by a sentence parsing tool (such as SceneGraph Parser) to obtain feature information in the training text. It can be understood that the feature information (label information, attribute information, relationship information) is automatically parsed from the existing training text, without the need for additional data annotation, and more visual semantic understanding tasks can be completed with relatively less computing power and data, which is highly efficient.

[0047] In some embodiments, when a training text is obtained, the text components of the training text are parsed, and based on the parsed text components, information related to the relationship between words corresponding to the text components is determined from the training image, and then label information (content words), attribute information (words that modify label information), relationship information (relationship words), etc. can be obtained from the training image, without limitation here.

[0048] In some embodiments, when a training text and a training image are obtained, it is possible to detect whether the training text satisfies a first preset condition, and to detect whether the training image satisfies a second preset condition, wherein, when it is detected that the training text satisfies the first preset condition and the training image satisfies the second preset condition, the training text can be parsed to obtain feature information in the training text. As an implementable manner, when it is detected that the text content of the training text is complete, it can be determined that the training text satisfies the first preset condition, and when it is detected that the image content of the training image is complete, it can be determined that the training image satisfies the second preset condition. As another implementable manner, when it is detected that the text content of the training text does not include illegal text, it can be determined that the training text satisfies the first preset condition, and when it is detected that the image content of the training image does not include illegal images, it can be determined that the training image satisfies the second preset condition.

[0049] Step S130: input the training text and the training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model, train the initial model, and obtain a visual semantic understanding model.

[0050] In this embodiment, an initial model can be selected for training. For example, a convolutional neural network (CNN) can be selected as the initial model for training. The initial model may include a backbone network and a detection head, wherein the input of the backbone network may include training text and training images, the output of the backbone network may be used as the input of the detection head, and the input of the detection head may also include label information, attribute information, and relationship information. The output of the detection head may include visual semantic understanding results. As a method, the backbone network may be composed of multiple layers of the same transformer encoder stacked together.

[0051] In some embodiments, when the label information, attribute information and relationship information of the training text are obtained, the training text, training image, label information, attribute information and relationship information can be input into the initial model together to train the initial model. As an implementable method, the training text and training image can be input into the backbone network of the initial model to participate in the forward calculation, and the label information, attribute information and relationship information can be input into the detection head of the initial model as the true value to participate in the loss function calculation, so that the visual semantic understanding model can be trained.

[0052] In some embodiments, for the backbone network, the input training text is segmented into a token sequence of length M, the input training image is first passed through the convolutional neural network, and then divided into N visual tokens, and these tokens are arranged together and input into the backbone network (such as Figure 2 As shown in the figure, position embedding and segment embedding are omitted. Among these tokens, there are two special tokens: [IMG] and [GLS]. [IMG] represents the image itself, and its corresponding embedding is the global convolutional feature of the image. [GLS] is added to the beginning of the text, and its role is the same as in BERT. Among them, the output part of the backbone network is M+N+2 hidden layer vectors, which can be regarded as feature vectors combined with context.

[0053] In some embodiments, when a feature vector is obtained as an output of the backbone network, the feature vector can be input into the detection head of the initial model as an input parameter of the detection head to obtain feature information output by the detection head based on the input feature vector. The feature information output by the detection head based on the input feature vector and the feature information as the true value are then used to calculate a loss function to optimize the trained model to obtain a visual semantic understanding model.

[0054] In some embodiments, when a visual semantic understanding model is obtained through training, the visual semantic understanding model can be uploaded to the cloud, which can communicate with multiple electronic devices. When an electronic device has a task of visual semantic understanding, the text to be inferred and the image to be inferred can be sent to the cloud to instruct the cloud to infer the text to be inferred and the image to be inferred through the visual semantic understanding model to obtain the corresponding reasoning results, and feed the reasoning results back to the electronic device. Based on this, by storing the visual semantic understanding model in the cloud, the storage space of the electronic device can be freed up and the storage pressure of the electronic device can be reduced.

[0055] In some embodiments, when a visual semantic understanding model is obtained through training, the visual semantic understanding model can be transplanted to an electronic device. When a visual semantic understanding task exists on an electronic device, the text to be inferred and the image to be inferred can be input into the local visual semantic understanding model to obtain the inference result output by the visual semantic understanding model. Based on this, by transplanting the visual semantic understanding model to an electronic device, the influence of the network on the visual semantic understanding task can be weakened, that is, even in the absence of a network, the visual semantic understanding task can still be performed normally.

[0056] In some embodiments, when a visual semantic understanding model is obtained, the accuracy of the visual semantic understanding model can also be verified, and it can be determined whether the output information of the visual semantic understanding model based on the input data meets the preset requirements. When the output information of the visual semantic understanding model based on the input data does not meet the preset requirements, the training data set can be re-collected to train the initial model, or multiple training data sets can be obtained to correct the visual semantic understanding model so that the error is as close to 0 as possible.

[0057] A model training method provided in an embodiment of the present application obtains training text and training images, wherein the training text is used to describe the content in the training image, parses the training text, obtains feature information in the training text, wherein the feature information includes at least one of label information, attribute information, and relationship information, inputs the training text and the training image into the backbone network of the initial model, and inputs the feature information into the detection head of the initial model, trains the initial model, and obtains a visual semantic understanding model, thereby using the feature information of the parsed training text as the input of the detection head of the model to supervise key information during the model training process, thereby improving the controllability and accuracy of the model training.

[0058] See also Figure 3 , Figure 3 The flowchart of the training method of the model provided in one embodiment of the present application is shown in FIG. Figure 3 The process shown is described in detail, and the training method of the model can specifically include the following steps:

[0059] Step S210: Acquire training text and training images, wherein the training text is used to describe the content in the training image.

[0060] Step S220: Parse the training text to obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information.

[0061] For the detailed description of step S210 to step S220, please refer to step S110 to step S120, which will not be repeated here.

[0062] Step S230: determine a training task, and determine a detection head of the initial model corresponding to the training task.

[0063] In this embodiment, the training tasks included in visual semantic understanding may include MLM training tasks, multi-label classification training tasks, character interaction recognition training tasks, and ITM training tasks. Among them, MLM (Masked Language Modeling) refers to predicting masked text based on contextual information and visual information, and ITM (Image-Text Matching) refers to determining whether an image and a text match. In this embodiment, the detection head of the initial model may include a detection head composed of a fully connected layer + softmax, and may include a detection head composed of a sigmoid, etc.

[0064] In some embodiments, the electronic device may be pre-set and store a first mapping relationship table, which may include multiple training tasks, multiple detection heads, and correspondences between multiple training tasks and multiple detection heads, wherein the correspondence may include that one training task may correspond to one detection head, or that multiple training tasks may correspond to one detection head, etc., which are not limited here. Therefore, in this embodiment, the required training task may be determined, and the detection head of the initial model corresponding to the training task may be determined based on the first mapping relationship table.

[0065] In some implementations, the reasoning requirements may be obtained, and the training tasks may be determined based on the reasoning requirements. For example, if the reasoning requirements are multi-label classification reasoning requirements, the training tasks may be determined to be multi-label classification training tasks; if the reasoning requirements are task interaction recognition reasoning requirements, the training tasks may be determined to be character interaction recognition training tasks; if the reasoning requirements are image-text retrieval reasoning requirements, the training tasks may be determined to be ITM training tasks; if the reasoning requirements are picture-speaking reasoning requirements, the training tasks may be determined to be MLM training tasks, etc., without limitation here.

[0066] In some implementations, training requirements may be obtained, and training tasks may be determined based on the training requirements, wherein the training requirements may be set by a user or may be automatically set by an electronic device or a server, which is not limited here.

[0067] The number of the training tasks may be one or more, which is not limited here.

[0068] In some implementations, if the training task is an MLM training task, a multi-label classification training task, or a person interaction recognition training task, the fully connected layer and the softmax layer corresponding to the training task can be determined as the detection head of the initial model. As an implementable method, the MLM training task, the multi-label classification training task, or the person interaction recognition training task is equivalent to a multi-classification problem, which requires a fully connected layer with an output dimension equal to the number of categories, followed by a softmax layer.

[0069] In some implementations, if the training task is an ITM training task, a sigmoid layer corresponding to the training task is determined as the detection head of the initial model. As an implementable approach, the ITM training task is essentially a binary classification task that needs to output a value and can be followed by a sigmoid layer.

[0070] Step S240: input the training text and the training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model corresponding to the training task, train the initial model, and obtain the visual semantic understanding model.

[0071] In some embodiments, when the label information, attribute information, and relationship information of the training text are obtained, the training text and training images can be input into the backbone network of the initial model, and the label information, attribute information, and relationship information can be input into the detection head of the initial model corresponding to the training task to train the initial model and obtain a visual semantic understanding model.

[0072] For example, assuming that the training tasks include training tasks 1, 2, and 3, and the detection head of the initial model corresponding to training task 1 is detection head 1, the detection head of the initial model corresponding to training task 2 is detection head 2, and the detection head of the initial model corresponding to training task 3 is detection head 3. Then, when the training task is training task 1, the training text and training image can be input into the backbone network of the initial model, and the feature information can be input into detection head 1, the initial model is trained to obtain a visual semantic understanding model; when the training tasks include training tasks 2 and 3, the training text and training image can be input into the backbone network of the initial model, and the feature information can be input into detection head 2 and detection head 3, the initial model is trained to obtain a visual semantic understanding model.

[0073] It is understandable that, for different training tasks, this embodiment can use the same backbone network and detection heads adapted to different training tasks for combined training to obtain a visual semantic understanding model. Therefore, when the visual semantic understanding model obtained through training is used to reason about tasks, the backbone network is universal, and the detection head can be selected and set according to different reasoning tasks, so that the trained visual semantic understanding model can be used to reason about multiple tasks.

[0074] The model training method provided in one embodiment of the present application obtains training text and training images, wherein the training text is used to describe the content in the training image, parses the training text, obtains feature information in the training text, and the feature information includes at least one of label information, attribute information, and relationship information, determines the training task, and determines the detection head of the initial model corresponding to the training task, inputs the training text and training image into the backbone network of the initial model, and inputs the feature information into the detection head of the initial model corresponding to the training task, trains the initial model, and obtains a visual semantic understanding model. Compared to Figure 1 The training method of the model shown in the figure, this embodiment also sets different detection heads to train the model according to different training tasks to obtain a unified visual semantic understanding model, which can have both practicality and academic significance.

[0075] See also Figure 4 , Figure 4 The flowchart of the training method of the model provided in one embodiment of the present application is shown in FIG. Figure 4 The process shown is described in detail, and the training method of the model can specifically include the following steps:

[0076] Step S310: Acquire training text and training images, wherein the training text is used to describe the content in the training image.

[0077] Step S320: Parse the training text to obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information.

[0078] For the detailed description of step S310 to step S320, please refer to step S110 to step S120, which will not be repeated here.

[0079] Step S330: determine a training task, and determine input parameters of the detection head of the initial model corresponding to the training task.

[0080] In this embodiment, the input parameters of the detection head of the initial model may include some or all of the output parameters of the backbone network connected thereto. For example, the output parameters of the backbone network include M+N+2 ([IMG] and [GLS]) hidden layer vectors (tokens), and the input parameters of the detection head of the initial model may include the hidden layer vectors corresponding to the masked tokens, the hidden layer vectors corresponding to [IMG], the hidden layer vectors corresponding to [GLS], etc., which are not limited here.

[0081] In some embodiments, the electronic device may be pre-set and store a second mapping relationship table, which may include multiple training tasks, multiple input parameters, and correspondences between multiple training tasks and multiple training parameters, wherein the correspondence may include that one training task may correspond to one input parameter, or that multiple training tasks may correspond to one detection head, etc., which are not limited here. Therefore, in this embodiment, the required training task may be determined, and the input parameters of the detection head of the initial model corresponding to the training task may be determined based on the second mapping relationship table.

[0082] In some embodiments, if the training task is an MLM training task, the input parameter of the detection head of the initial model corresponding to the training task is determined to be the hidden layer vector corresponding to the masked token. As an implementable manner, the MLM training task belongs to the classification of the token level, and therefore, the input to the detection head of the initial model is the hidden layer vector corresponding to the masked token.

[0083] In some embodiments, if the training task is a multi-label classification training task or a person interaction recognition training task, the input parameter of the detection head of the initial model corresponding to the training task is determined to be the hidden layer vector corresponding to the training image. As an implementable method, the multi-label classification training task and the person interaction recognition training task belong to the classification at the sequence level, so the hidden layer vector corresponding to [IMG] is input to the detection head of the multi-label classification training task or the task interaction recognition training task.

[0084] In some implementations, if the training task is an ITM training task, the input parameter of the detection head of the initial model corresponding to the training task is determined to be the hidden vector of the training text. As an implementable method, the ITM training task belongs to the classification of the sequence level, so the hidden vector corresponding to [GLS] is input to the detection head of the ITM training task.

[0085] Step S340: input the training text and the training image into the backbone network of the initial model to obtain multiple hidden layer vectors output by the backbone network.

[0086] In this embodiment, when the training text and the training image are obtained, the training text and the training image can be input into the backbone network of the initial model to obtain the multi-layer hidden layer vectors input into the backbone network.

[0087] As a method, when the training text and training image are input into the backbone network, the input training text is segmented into a token sequence of length M, and the input training image is first passed through the convolutional neural network and then divided into N visual tokens. These tokens are arranged together and input into the backbone network, among which there are two special tokens: [IMG] and [GLS]. Based on the input tokens, the output part of the backbone network is M+N+2 hidden layer vectors.

[0088] Step S350: Determine a hidden layer vector corresponding to the input parameter from the multiple hidden layer vectors.

[0089] In this embodiment, when multiple hidden layer vectors output by the backbone network are obtained, the hidden layer vector corresponding to the input parameter can be determined from the multiple hidden layer vectors. As one way, when multiple hidden layer vectors output by the backbone network are obtained, the multiple hidden layer vectors can be traversed to determine the hidden layer vector corresponding to the input parameter from the multiple hidden layer vectors. As another way, when multiple hidden layer vectors output by the backbone network are obtained, a hidden layer vector can be randomly extracted from the multiple hidden layer vectors to determine whether it is a hidden layer vector corresponding to the input parameter until the corresponding hidden layer vector is determined therefrom.

[0090] Among them, when the training task is determined to be an ITM training task and the input parameter is the hidden vector corresponding to [GLS], the hidden vector corresponding to [GLS] can be obtained from multiple hidden vectors; when the training task is determined to be a multi-label classification training task or a person interaction recognition training task, the input parameter is the hidden vector corresponding to [IMG], the hidden vector corresponding to [IMG] can be obtained from multiple hidden vectors; when the training task is determined to be an MLM training task, the hidden vector corresponding to the masked token can be obtained from multiple hidden vectors.

[0091] Step S360: input the hidden layer vector corresponding to the input parameter into the detection head of the initial model, train the initial model, and obtain the visual semantic understanding model.

[0092] In this embodiment, when the hidden layer vector corresponding to the input parameter is determined, the hidden layer vector corresponding to the input parameter can be input into the detection head of the initial model, so that the initial model can be trained by training the detection head of the initial model to obtain a visual semantic understanding model.

[0093] In some embodiments, when the hidden layer vector corresponding to the input parameter is determined, the hidden layer vector and feature information (label information, attribute information, and relationship information) corresponding to the input parameter can be input into the detection head of the initial model to train the initial model and obtain a visual semantic understanding model.

[0094] In some embodiments, feature information extracted from the training text can be used as the true value. When the hidden vector corresponding to the input parameter is determined, the hidden vector corresponding to the input parameter can be input into the detection head of the initial model to obtain feature information output by the detection head of the initial model. Then, based on the feature information as the true value and the feature information output by the detection head of the initial model, the initial model is optimized to obtain a visual semantic understanding model.

[0095] The model training method provided in one embodiment of the present application obtains training text and training images, wherein the training text is used to describe the content in the training image, parses the training text, obtains feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information, determines the training task, and determines the input parameters of the detection head of the initial model corresponding to the training task, inputs the training text and the training image into the backbone network of the initial model, obtains multiple hidden layer vectors output by the backbone network, determines the hidden layer vector corresponding to the input parameter from the multiple hidden layer vectors, inputs the hidden layer vector corresponding to the input parameter into the detection head of the initial model, trains the initial model, and obtains a visual semantic understanding model. Compared to Figure 1 The training method of the model shown in the figure, this embodiment also sets different input parameters for model training according to different training tasks to obtain a unified visual semantic understanding model, which can have both practicality and academic significance.

[0096] See also Figure 5 , Figure 5 The flowchart of the training method of the model provided in one embodiment of the present application is shown in FIG. Figure 5 The process shown is described in detail, and the training method of the model can specifically include the following steps:

[0097] Step S410: Acquire training text and training images, wherein the training text is used to describe the content in the training image.

[0098] Step S420: Parse the training text to obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information.

[0099] Step S430: input the training text and the training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model, train the initial model, and obtain a visual semantic understanding model.

[0100] For the detailed description of step S410 to step S130 , please refer to step S110 to step S130 , which will not be repeated here.

[0101] Step S440: Acquire the text to be inferred and the image to be inferred, wherein the text to be inferred is used to describe the content in the image to be inferred.

[0102] In this embodiment, a text to be inferred and an image to be inferred may be obtained, wherein the text to be inferred is used to describe the content in the image to be inferred. For example, the text to be inferred may include "there is a woman on the beach", and the image to be inferred may include a beach.

[0103] In some implementations, the number of the text to be inferred and the image to be inferred may be one or more, which is not limited herein.

[0104] In some embodiments, the text to be inferred and the image to be inferred can be obtained from a server via a network, for example, the text to be inferred and the image to be inferred can be obtained from a server via a wireless network, the text to be inferred and the image to be inferred can be obtained from a server via a wired network, the text to be inferred and the image to be inferred can be obtained from a server via a data network, etc.; the text to be inferred and the image to be inferred can be obtained from other devices via a network, for example, the text to be inferred and the image to be inferred can be obtained from other devices via a wireless network, the text to be inferred and the image to be inferred can be obtained from other devices via a wired network, the text to be inferred and the image to be inferred can be obtained from other devices via a data network, etc.; the text to be inferred and the image to be inferred can also be obtained directly from the local electronic device, etc., which is not limited here.

[0105] Step S450: Determine the task to be inferred.

[0106] In some implementations, the task to be reasoned may include at least one of a multi-label classification reasoning task, a character interaction recognition reasoning task, a picture-text retrieval reasoning task, and a picture-speaking reasoning task.

[0107] In some implementations, the reasoning requirements may be obtained, and the task to be reasoned may be determined based on the reasoning requirements. For example, if the reasoning requirement is a multi-label classification reasoning requirement, the task to be reasoned may be determined to be a multi-label classification reasoning task; if the reasoning requirement is a task interaction recognition reasoning requirement, the task to be reasoned may be determined to be a character interaction recognition reasoning task; if the reasoning requirement is a picture-text retrieval reasoning requirement, the task to be reasoned may be determined to be a picture-text retrieval reasoning task; if the reasoning requirement is a picture-speaking reasoning requirement, the task to be reasoned may be determined to be a picture-speaking reasoning task, etc., and no limitation is made here.

[0108] As an implementable method, the visual semantic understanding model can be transplanted to an electronic device after training. Accordingly, the electronic device can provide a variety of choices of tasks to be inferred, and the electronic device can determine the task to be inferred that needs to be executed at this time from the multiple tasks to be inferred based on the task selection instruction input by the user. For example, the electronic device can display a selection control for the task to be inferred, and the user can trigger the task selection instruction by operating the selection control. Accordingly, the electronic device can respond to the task selection instruction and determine the task to be inferred that needs to be executed at this time from the multiple tasks to be inferred. For another example, the electronic device can receive voice information input by the user, and parse the voice information to obtain the task selection instruction carried in the voice information. Accordingly, the electronic device can respond to the task selection instruction and determine the task to be inferred that needs to be executed at this time from the multiple tasks to be inferred.

[0109] As an implementable method, the visual semantic understanding model can be transplanted to an electronic device after training. Accordingly, the electronic device can determine the task to be inferred that needs to be performed at this time from a variety of tasks to be inferred based on the scene it is in. For example, the location of the electronic device can be obtained, and based on the location of the electronic device, the task to be inferred that needs to be performed at this time can be determined from a variety of tasks to be inferred. For another example, the surrounding image of the electronic device can be obtained, and based on the surrounding image of the electronic device, the task to be inferred that needs to be performed at this time can be determined from a variety of tasks to be inferred.

[0110] Step S460: input the text to be inferred and the image to be inferred into the visual semantic understanding model, and obtain the inference result related to the task to be inferred output by the visual semantic understanding model.

[0111] In this embodiment, when the task to be reasoned is determined, the text to be reasoned and the image to be reasoned can be input into the visual semantic understanding model to obtain the reasoning result related to the reasoning task output by the visual semantic understanding model. As an implementable method, when the task to be reasoned is determined, the text to be reasoned and the image to be reasoned can be input into the backbone network in the visual semantic understanding model to obtain the corresponding hidden layer vector output by the backbone network in the visual semantic understanding model, and then the corresponding hidden layer vector output by the backbone network is input into the detection head corresponding to the task to be reasoned in the visual semantic understanding model to obtain the reasoning result related to the task to be reasoned output by the detection head corresponding to the task to be reasoned in the visual semantic understanding model.

[0112] In some embodiments, if the task to be inferred is a multi-label classification reasoning task, the text to be inferred and the image to be inferred can be input into the visual semantic understanding model to obtain the reasoning result related to the multi-label classification reasoning task output by the visual semantic understanding model. As a method, the text to be inferred and the image to be inferred can be input into the backbone network in the visual semantic understanding model to obtain the corresponding hidden layer vector output by the backbone network in the visual semantic understanding model, and the hidden layer vector corresponding to [IMG] in the corresponding hidden layer vector output by the backbone network is input into the fully connected layer and the softmax layer in the visual semantic understanding model to obtain the reasoning result related to the multi-label classification reasoning task.

[0113] In some embodiments, if the task to be inferred is a character interaction recognition reasoning task, the text to be inferred and the image to be inferred can be input into the visual semantic understanding model to obtain the reasoning result related to the character interaction recognition reasoning task output by the visual semantic understanding model. As a method, the text to be inferred and the image to be inferred can be input into the backbone network in the visual semantic understanding model to obtain the corresponding hidden vector output by the backbone network in the visual semantic understanding model, and the hidden vector corresponding to [IMG] in the corresponding hidden vector output by the backbone network is input into the fully connected layer and the softmax layer in the visual semantic understanding model to obtain the reasoning result related to the character interaction recognition reasoning task.

[0114] In some embodiments, if the task to be inferred is a picture-text retrieval task, the text to be inferred and the image to be inferred can be input into the visual semantic understanding model to obtain the reasoning result related to the picture-text retrieval task output by the visual semantic understanding model. As a method, the text to be inferred and the image to be inferred can be input into the backbone network in the visual semantic understanding model to obtain the corresponding hidden vector output by the backbone network in the visual semantic understanding model, and the hidden vector corresponding to [CLS] in the corresponding hidden vector output by the backbone network is input into the sigmoid layer in the visual semantic understanding model to obtain the reasoning result related to the picture-text retrieval reasoning task. Among them, for the picture-text retrieval reasoning task, the hidden vectors corresponding to [IMG] and [CLS] can be used as the global features of the image and text respectively, and these features are used to calculate the distance between the image and the text, and then the retrieval results are obtained by sorting according to the distance.

[0115] In some embodiments, if the task to be inferred is a picture-speaking reasoning task, the text to be inferred and the image to be inferred can be input into the visual semantic understanding model to obtain the reasoning result related to the picture-speaking reasoning task output by the visual semantic understanding model. As a method, the text to be inferred and the image to be inferred can be input into the backbone network in the visual semantic understanding model to obtain the corresponding hidden vector output by the backbone network in the visual semantic understanding model, and the hidden vector corresponding to the masked token in the corresponding hidden vector output by the backbone network is input into the fully connected layer and the softmax layer in the visual semantic understanding model to obtain the reasoning result related to the picture-speaking reasoning task. Among them, for the picture-speaking reasoning task, the detection head of the MLM training task is used to predict the next word. At the beginning, a special mark indicating the beginning of the sentence is input to predict the first word. Then, the obtained word sequence is input each time to predict the next word, and this cycle is repeated until the visual semantic understanding model outputs a special mark indicating the end of the sentence.

[0116] See also Figure 6 , Figure 6 A schematic diagram showing the framework of the visual semantic understanding model provided by the embodiment of the present application is shown. Figure 6As shown, the visual semantic understanding model can be trained through different training tasks, and can be used to perform different reasoning tasks after the training is completed. In particular, when images and texts are obtained, the texts can be parsed to obtain the labels, attributes, and relationships in the texts, and then the images, texts, labels, attributes, and relationships are input into the model for training, such as MLM training tasks, ITM training tasks, multi-label classification training tasks, and character interaction recognition training tasks. Subsequently, the trained visual semantic understanding model can be used to perform reasoning tasks for the input graphics and texts, such as multi-label classification reasoning tasks, character interaction recognition reasoning tasks, image-text retrieval reasoning tasks, and picture-speaking reasoning tasks.

[0117] The training method of the model provided in one embodiment of the present application obtains training text and training images, wherein the training text is used to describe the content in the training image, parses the training text, obtains feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information, inputs the training text and training images into the backbone network of the initial model, and inputs the feature information into the detection head of the initial model, trains the initial model, obtains the visual semantic understanding model, obtains the text to be inferred and the image to be inferred, wherein the text to be inferred is used to describe the content in the image to be inferred, determines the task to be inferred, inputs the text to be inferred and the image to be inferred into the visual semantic understanding model, and obtains the reasoning result related to the task to be inferred output by the visual semantic understanding model. Compared to Figure 1 The training method of the model shown in the figure, this embodiment also uses the visual semantic understanding model to reason on the selected task to be reasoned to obtain the reasoning result, thereby improving the application scope of the model and enhancing the user experience.

[0118] See also Figure 7 , Figure 7 The module block diagram of the training device of the model provided by an embodiment of the present application is shown. Figure 7 The block diagram shown in the figure is used to illustrate that the model training device 200 includes: an information acquisition module 210, an information parsing module 220 and a model training module 230, wherein:

[0119] The information acquisition module 210 is used to acquire training text and training images, wherein the training text is used to describe the content in the training image.

[0120] The information parsing module 220 is used to parse the training text to obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information.

[0121] The model training module 230 is used to input the training text and the training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model, train the initial model, and obtain a visual semantic understanding model.

[0122] Furthermore, the model training module 230 includes: a detection head determination submodule and a first model training submodule, wherein:

[0123] The detection head determination submodule is used to determine a training task and determine the detection head of the initial model corresponding to the training task.

[0124] Further, the detection head determination submodule includes: a first detection head determination unit and a second detection head determination unit, wherein:

[0125] The first detection head determination unit is used to determine the fully connected layer and softmax layer corresponding to the training task as the detection head of the initial model if the training task is an MLM training task, a multi-label classification training task or a person interaction recognition training task.

[0126] The second detection head determination unit is used to determine the sigmoid layer corresponding to the training task as the detection head of the initial model if the training task is an ITM training task.

[0127] The first model training submodule is used to input the training text and the training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model corresponding to the training task, train the initial model, and obtain the visual semantic understanding model.

[0128] Furthermore, the model training module 230 includes: an input parameter determination submodule, a hidden layer vector acquisition submodule, a hidden layer vector screening submodule and a second model training submodule, wherein:

[0129] The input parameter determination submodule is used to determine a training task and determine the input parameters of the detection head of the initial model corresponding to the training task.

[0130] Furthermore, the input parameter determination submodule includes: a first input parameter determination unit, a second input parameter determination unit and a third input parameter determination unit, wherein:

[0131] The first input parameter determination unit is used to determine, if the training task is an MLM training task, that the input parameter of the detection head of the initial model corresponding to the training task is a hidden layer vector corresponding to the masked token.

[0132] The second input parameter determination unit is used to determine that the input parameter of the detection head of the initial model corresponding to the training task is the hidden layer vector corresponding to the training image if the training task is a multi-label classification training task or a person interaction recognition training task.

[0133] The third input parameter determination unit is used to determine, if the training task is an ITM training task, that the input parameter of the detection head of the initial model corresponding to the training task is a hidden layer vector of the training text.

[0134] The hidden layer vector acquisition submodule is used to input the training text and the training image into the backbone network of the initial model to obtain multiple hidden layer vectors output by the backbone network.

[0135] The hidden layer vector screening submodule is used to determine a hidden layer vector corresponding to the input parameter from the multiple hidden layer vectors.

[0136] The second model training submodule is used to input the hidden layer vector corresponding to the input parameter into the detection head of the initial model, train the initial model, and obtain the visual semantic understanding model.

[0137] Furthermore, the model training device 200 further includes: a reasoning information acquisition module, a to-be-reasoned task determination module, and a reasoning result acquisition module, wherein:

[0138] The inference information acquisition module is used to acquire the text to be inferred and the image to be inferred, wherein the text to be inferred is used to describe the content in the image to be inferred.

[0139] The module for determining tasks to be inferred is used to determine tasks to be inferred.

[0140] The inference result acquisition module is used to input the text to be inferred and the image to be inferred into the visual semantic understanding model to obtain the inference result related to the task to be inferred output by the visual semantic understanding model.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0142] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0143] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0144] See also Figure 8 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0145] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0146] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0147] See also Fig. 9 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0148] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 300 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0149] In summary, the model training method, device, electronic device and storage medium provided in the embodiments of the present application obtain training text and training images, wherein the training text is used to describe the content in the training image, parse the training text, and obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information, input the training text and training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model, train the initial model, and obtain a visual semantic understanding model, so as to realize supervision of key information during the model training process by parsing the feature information of the training text as the input of the detection head of the model, thereby improving the controllability and accuracy of the model training.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire training text and training images, wherein the training text is used to describe the content in the training image; Parsing the training text to obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information, and relationship information; The training text and the training image are input into the backbone network of the initial model, and the feature information is input into the detection head of the initial model, and the initial model is trained to obtain a visual semantic understanding model.

2. The method according to claim 1, characterized in that The step of inputting the training text and the training image into the backbone network of the initial model, inputting the feature information into the detection head of the initial model, training the initial model, and obtaining a visual semantic understanding model includes: Determining a training task, and determining a detection head of the initial model corresponding to the training task; The training text and the training image are input into the backbone network of the initial model, and the feature information is input into the detection head of the initial model corresponding to the training task, the initial model is trained, and the visual semantic understanding model is obtained.

3. The method according to claim 2, characterized in that The determining of the detection head of the initial model corresponding to the training task includes: If the training task is an MLM training task, a multi-label classification training task, or a person interaction recognition training task, determining a fully connected layer and a softmax layer corresponding to the training task as the detection head of the initial model; or If the training task is an ITM training task, a sigmoid layer corresponding to the training task is determined as the detection head of the initial model.

4. The method according to claim 1, characterized in that The step of inputting the training text and the training image into the backbone network of the initial model, inputting the feature information into the detection head of the initial model, training the initial model, and obtaining a visual semantic understanding model includes: Determine a training task, and determine input parameters of the detection head of the initial model corresponding to the training task; Inputting the training text and the training image into the backbone network of the initial model to obtain a plurality of hidden layer vectors output by the backbone network; Determine a hidden layer vector corresponding to the input parameter from the multiple hidden layer vectors; The hidden layer vector corresponding to the input parameter is input into the detection head of the initial model, the initial model is trained, and the visual semantic understanding model is obtained.

5. The method according to claim 4, characterized in that The step of determining the input parameters of the detection head of the initial model corresponding to the training task includes: If the training task is an MLM training task, determining that the input parameter of the detection head of the initial model corresponding to the training task is a hidden layer vector corresponding to the masked token; or If the training task is a multi-label classification training task or a person interaction recognition training task, determining that the input parameter of the detection head of the initial model corresponding to the training task is a hidden layer vector corresponding to the training image; or If the training task is an ITM training task, the input parameter of the detection head of the initial model corresponding to the training task is determined to be the hidden layer vector of the training text.

6. The method according to any one of claims 1 to 5, characterized in that: After obtaining the visual semantic understanding model, the method further includes: Acquire a text to be inferred and an image to be inferred, wherein the text to be inferred is used to describe the content in the image to be inferred; Determine the task to be reasoned; The text to be inferred and the image to be inferred are input into the visual semantic understanding model to obtain the inference result related to the task to be inferred output by the visual semantic understanding model.

7. The method according to claim 6, characterized in that The task to be inferred includes at least one of a multi-label classification reasoning task, a character interaction recognition reasoning task, a picture-text retrieval reasoning task, and a picture-speaking reasoning task.

8. A model training device, characterized in that: The device comprises: An information acquisition module, used to acquire training text and training images, wherein the training text is used to describe the content in the training image; An information parsing module, used to parse the training text to obtain feature information in the training text, wherein the feature information includes at least one of label information, attribute information and relationship information; The model training module is used to input the training text and the training image into the backbone network of the initial model, and input the feature information into the detection head of the initial model, train the initial model, and obtain a visual semantic understanding model.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, and the memory stores instructions. When the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images