Image recognition model training method and device, electronic equipment and storage medium

Abstract

The invention discloses an image recognition model training method and device, electronic equipment and a storage medium, and relates to the technical field of feature learning, and the main technical scheme comprises the steps: obtaining training question and answer data; training a visual universal model according to the training question and answer data and the training image data to obtain a prediction answer, generated by the visual universal model, of the training question and answer data; and calculating a loss function of the visual universal model according to the standard answer and the predicted answer, and performing parameter adjustment on the visual universal model according to the loss function. According to the scheme that various tasks are unified into question-answer data pairs, various visual task data are uniformly trained through a language interface, so that a new visual universal model is obtained, a network has better visual-language space alignment capability, visual information of various levels can be better processed and captured, and the visual effect is improved. And the capability and the effect of a mainstream multi-modal large language model can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of feature learning technology, and in particular to a training method and device for an image recognition model, an electronic device, and a storage medium. Background Art

[0002] The emergence of large-scale multimodal language models is a significant milestone in the development of artificial intelligence. This extends the capabilities of language models beyond text comprehension to include the ability to summarize and interpret visual data. This integration has opened up new areas of computer vision, enabling models to process and analyze information cross-modally, much like humans. Through training, an increasing number of models are able to understand and combine the content of text and images. The quality of visual tokens, which represent the input visual data, plays a crucial role in the overall effectiveness of large-scale multimodal language models. High-quality visual tokens enable large-scale multimodal language models to more accurately perceive and interact with visual elements, enabling more nuanced and contextually relevant multimodal understanding.

[0003] Currently, visual instruction fine-tuning has become a popular and efficient method for training multimodal large language models. This method combines a pre-trained language model and a pre-trained visual encoder, and fine-tunes the model using visual and language instruction data. Most existing works use CLIP (Contrastive Language-Image Pre-training, a visual encoder based on contrastive learning) as a visual encoder to generate visual tokens for large language models, because the training process of CLIP requires learning from a large number of image-text pair datasets, which can naturally align visual data with the language embedding space. Although this type of visual encoder pre-trained by contrastive learning can achieve fine-tuning of modal large oracle models, this type of visual encoding lacks the fine-grained details required to solve more complex visual understanding problems. It cannot fully represent tasks related to object localization and recognition, or understand low-level structural elements of images. Summary of the Invention

[0004] The present disclosure provides a training method, apparatus, electronic device, and storage medium for an image recognition model, primarily to address the inability to adequately characterize tasks related to object location and recognition, or to understand low-level structural elements of an image.

[0005] According to a first aspect of the present disclosure, a method for training an image recognition model is provided, comprising:

[0006] Obtaining training question-answering data; wherein the training question-answering data is generated based on the training image data; the training question-answering data includes questions and standard answers to the questions;

[0007] Training a general vision model based on the training question-answer data and the training image data to obtain a predicted answer to the training question-answer data generated by the general vision model;

[0008] A loss function of the visual general model is calculated according to the standard answer and the predicted answer, and parameters of the visual general model are adjusted according to the loss function.

[0009] Optionally, the training question-and-answer data includes visual-to-language conversion data, and the training of the general vision model based on the training question-and-answer data and the training image data to obtain a predicted answer to the training question-and-answer data generated by the general vision model further includes:

[0010] The training image data is input into the general vision model to obtain a predicted text description of the training image output by the general vision model.

[0011] Optionally, the training question-answering data includes standard text descriptions of training image data, and calculating a loss function of the general vision model based on the standard answers and the predicted answers, and adjusting parameters of the general vision model based on the loss function includes:

[0012] A loss function is calculated according to the predicted text description and the standard text description, and the visual general model is trained according to the loss function.

[0013] Optionally, the training question-and-answer data includes an object recognition task, and the training of the general vision model based on the training question-and-answer data and the training image data to obtain a predicted answer to the training question-and-answer data generated by the general vision model includes:

[0014] Training the general vision model according to an object recognition task in the training question-and-answer data; wherein the object recognition task includes at least one of object name prediction, quantity counting, area description, listing target locations, and question-and-answering within a bounding box;

[0015] Generate a prediction token based on the concatenation of the image query vector and the language embedding vector in the general vision model;

[0016] Obtain a dense mask matrix of the training image data, divide the dense mask matrix into a preset number of matrix blocks, align the dense mask matrix into a language embedding space through a multilayer perceptron, and convert the training image data back to the original format of bounding boxes and dense masks based on the embedding labels.

[0017] Optionally, calculating a loss function of the general vision model according to the standard answer and the predicted answer, and adjusting parameters of the general vision model according to the loss function further includes:

[0018] Supervising the bounding box predicted by the general vision model based on L1 loss and generalized intersection-over-union loss;

[0019] Calculating the sum of absolute differences between the predicted depth mask and the true depth mask using L1 loss, and optimizing the general vision model based on the calculation result;

[0020] A similarity between the predicted token and a next token in the true sentence is determined based on a cross-entropy loss, and the visual general model is optimized according to the similarity.

[0021] Optionally, after calculating the loss function of the general vision model according to the standard answer and the predicted answer, and adjusting parameters of the general vision model according to the loss function, the method further includes:

[0022] After the number of training times reaches a preset threshold, the training of the general vision model is stopped; or,

[0023] After the loss function is less than a preset loss function threshold, the training of the general vision model is stopped.

[0024] Optionally, after stopping the training of the general vision model, the method further includes:

[0025] The visual general model is used as the visual encoder of a multimodal large language model to perform visual understanding of the image to be recognized.

[0026] According to a second aspect of the present disclosure, there is provided a training device for an image recognition model, comprising:

[0027] An acquisition unit, configured to acquire training question-answering data; wherein the training question-answering data is generated based on the training image data; and the training question-answering data includes questions and standard answers to the questions;

[0028] a training unit, configured to train a general vision model based on the training question-answer data and the training image data, and obtain a predicted answer to the training question-answer data generated by the general vision model;

[0029] A parameter adjustment unit is used to calculate the loss function of the visual general model based on the standard answer and the predicted answer, and to adjust the parameters of the visual general model based on the loss function.

[0030] Optionally, the training question-answering data includes visual-to-language conversion data, and the training unit is further configured to:

[0031] The training image data is input into the general vision model to obtain a predicted text description of the training image output by the general vision model.

[0032] Optionally, the training question-answering data includes a standard text description of the training image data, and the parameter adjustment unit is further configured to:

[0033] A loss function is calculated according to the predicted text description and the standard text description, and the visual general model is trained according to the loss function.

[0034] Optionally, the training question-answering data includes an object recognition task, and the training unit includes:

[0035] A first training module trains the general vision model based on object recognition tasks in the training question-and-answer data, wherein the object recognition tasks include at least one of object name prediction, quantity counting, region description, enumeration of target locations, and question-and-answering within a bounding box;

[0036] A second training module is configured to generate a prediction token based on the concatenation of the image query vector and the language embedding vector in the general vision model;

[0037] The third training module is configured to obtain a dense mask matrix of the training image data, divide the dense mask matrix into a preset number of matrix blocks, align the dense mask matrix into a language embedding space through a multi-layer perceptron, and convert the training image data back into the original format of bounding boxes and dense masks based on the embedding tags.

[0038] Optionally, the parameter adjustment unit is further configured to:

[0039] Supervising the bounding box predicted by the general vision model based on L1 loss and generalized intersection-over-union loss;

[0040] Calculating the sum of absolute differences between the predicted depth mask and the true depth mask using L1 loss, and optimizing the general vision model based on the calculation result;

[0041] A similarity between the predicted token and a next token in the true sentence is determined based on a cross-entropy loss, and the visual general model is optimized according to the similarity.

[0042] Optionally, the device further includes:

[0043] a stopping training unit, configured to stop training the visual general model after the parameter adjustment unit calculates the loss function of the visual general model according to the standard answer and the predicted answer and adjusts the parameters of the visual general model according to the loss function, and after the number of training times reaches a preset number threshold; or

[0044] The training stop unit is further used to stop training the general vision model after the loss function is less than a preset loss function threshold.

[0045] Optionally, the device further includes:

[0046] A visual understanding unit is used to use the visual general model as a visual encoder of a multimodal large language model to perform visual understanding of the image to be recognized after the stop training unit stops training the visual general model.

[0047] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0048] at least one processor; and

[0049] a memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0051] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0052] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0053] The training method, device, electronic device and storage medium of the image recognition model provided by the present disclosure have the following main technical solutions: obtaining training question and answer data; wherein the training question and answer data is generated based on training image data; the training question and answer data includes questions and standard answers to the questions; training a general visual model based on the training question and answer data and the training image data to obtain predicted answers to the training question and answer data generated by the general visual model; calculating the loss function of the general visual model based on the standard answers and the predicted answers, and adjusting the parameters of the general visual model based on the loss function. Compared with the related art, the embodiment of the present application unifies multiple tasks into question-answer data pairs, and uniformly trains multiple visual task data through a language interface, thereby obtaining a new general visual model. The network has better visual-language space alignment capabilities, and can better process and capture visual information at various levels, which can effectively improve the capabilities and effects of mainstream multimodal large language models.

[0054] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0056] Figure 1 A flowchart of a method for training an image recognition model provided by an embodiment of the present disclosure;

[0057] Figure 2 A flowchart of a method for training an image recognition model provided by an embodiment of the present disclosure;

[0058] Figure 3 A flowchart of a method for training an image recognition model provided by an embodiment of the present disclosure;

[0059] Figure 4 A schematic diagram of the structure of a training device for an image recognition model provided in an embodiment of the present disclosure;

[0060] Figure 5 A schematic diagram of the structure of a training device for an image recognition model provided in an embodiment of the present disclosure;

[0061] Figure 6 A schematic block diagram of an exemplary electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0063] The following describes the image recognition model training method, device, electronic device and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0064] Figure 1 A flowchart of a method for training an image recognition model provided in an embodiment of the present disclosure.

[0065] like Figure 1 As shown, the method comprises the following steps:

[0066] Step 101, obtain training question-answering data; wherein, the training question-answering data is generated based on training image data; the training question-answering data includes questions and standard answers to the questions.

[0067] In some embodiments, question-answer data pairs based on various visual tasks are constructed. A large amount of question-answer data is constructed and acquired using datasets such as visual question answering, object detection and recognition, and dense prediction. Specifically, this data may include data on object recognition, scene understanding, image segmentation, and other types. The specific embodiments of this application do not limit the data types. The resulting data can be used to train general visual models for various visual tasks.

[0068] Step 102: Train the general vision model based on the training question-answer data and the training image data to obtain a predicted answer to the training question-answer data generated by the general vision model.

[0069] In some embodiments, multi-task learning can be configured, where the model not only answers questions in the visual question-answering task, but also completes other visual tasks such as object detection, recognition, and dense prediction. This allows the model to learn richer feature representations, thereby improving its performance on different tasks. Specifically, all question-answer pairs, including vision-language, object recognition, and dense prediction, are mixed during training.

[0070] Step 103: Calculate the loss function of the visual general model according to the standard answer and the predicted answer, and adjust the parameters of the visual general model according to the loss function.

[0071] In some embodiments, due to the different training tasks involved in the training process, corresponding loss functions are designed for different tasks. For example, for visual question answering tasks, the similarity between the language model output and the true answer can be supervised based on binary cross-entropy loss; for object detection tasks, bounding box prediction can be supervised based on L1 loss and gIoU loss. These loss functions work together during the model training process to optimize its performance.

[0072] At each iteration, a loss value is calculated based on the model's predicted answer and the ground truth. The loss value reflects the model's current performance level, that is, how close it is to the correct answer. After calculating the loss value, the backpropagation algorithm is used to calculate the gradient. By calculating the derivative of the loss function with respect to each parameter, the backpropagation algorithm determines how to adjust the parameters to minimize the loss function. This propagates the error signal forward from the output layer and calculates the gradient for each layer.

[0073] The training method of the image recognition model provided by the present disclosure includes the following main technical solutions: obtaining training question-answering data; wherein, the training question-answering data is generated based on training image data; the training question-answering data includes questions and standard answers to the questions; training a general visual model based on the training question-answering data and the training image data to obtain predicted answers to the training question-answering data generated by the general visual model; calculating the loss function of the general visual model based on the standard answers and the predicted answers, and adjusting the parameters of the general visual model based on the loss function. Compared with the related art, the embodiment of the present application unifies multiple tasks into question-answer data pairs, and uniformly trains multiple visual task data through a language interface, thereby obtaining a new general visual model. The network has better visual-language space alignment capabilities, and can better process and capture visual information at various levels, which can effectively improve the capabilities and effects of mainstream multimodal large language models.

[0074] In some embodiments, the training question-and-answer data is mainly divided into three aspects: vision-language tasks, object recognition tasks, and dense prediction tasks. When training based on the vision-language conversion data in the training question-and-answer data, please refer to Figure 2. Figure 2 is a flowchart of a training method for an image recognition model provided in an embodiment of the present disclosure, including:

[0075] Step 201: input the training image data into the general vision model to obtain a predicted text description of the training image output by the general vision model.

[0076] In some embodiments, the parameters of the visual backbone are first fixed, and a Transformer-based Q-former, an image query vector, and a language decoder with a low-rank adaptation module are trained. The main focus of this stage is the vision-language task, encouraging the model to form a connection between the visual input and its language counterpart, thereby achieving preliminary cross-modal alignment.

[0077] Step 202 : Calculate a loss function based on the predicted text description and the standard text description, and train the general vision model based on the loss function.

[0078] After obtaining the predicted text description output by the model, we need to evaluate the degree of difference between it and the standard answer. Define a loss function to measure the gap between the two. Commonly used loss functions include binary cross entropy loss, mean square error loss, etc. The embodiment of the present application uses binary cross entropy loss as a measurement indicator. Specifically, the predicted text description is compared with the standard answer word by word, and the difference in the probability distribution of each word is calculated, and the differences of all words are added up to obtain the total loss value. Finally, the backpropagation algorithm is used to calculate the gradient, and the model parameters are updated through optimization algorithms (such as stochastic gradient descent, Adam, etc.) to minimize the loss function.

[0079] In some embodiments, training based on object recognition tasks and dense prediction task data in the training question-and-answer data includes the following steps. See Figure 3, which is a flow chart of a training method for an image recognition model provided by an embodiment of the present disclosure, including:

[0080] Step 301: Training the general vision model based on object recognition tasks in the training question-answering data; wherein the object recognition tasks include at least one of object name prediction, quantity counting, region description, listing target locations, and question-answering within a bounding box;

[0081] In some embodiments, the visual backbone is unfrozen and the entire model is trained jointly. This holistic approach enables the general vision model to refine its understanding of visual content in more detail, integrating object recognition and dense prediction tasks.

[0082] In some embodiments, the tasks in the object recognition task cover multiple levels from basic object recognition to more complex scene understanding, aiming to comprehensively improve the model's visual understanding and description capabilities.

[0083] Step 302: Generate a prediction token based on the concatenation of the image query vector and the language embedding vector in the general vision model.

[0084] Specifically, we combine the feature vector extracted from the image (i.e., the image query vector) with the question text encoded using natural language processing (i.e., the language embedding vector) to form a unified feature representation. Natural language processing techniques can be used to convert the question text into a series of word vectors that capture the meaning and context of the question. The image feature vector and word vectors are then concatenated to form a new feature representation. This new feature representation incorporates both the visual information of the image and the semantic information of the question, enabling the model to better understand the question and provide accurate responses.

[0085] Step 303: Obtain a dense mask matrix of the training image data, divide the dense mask matrix into a preset number of matrix blocks, align the dense mask matrix into a language embedding space through a multi-layer perceptron, and convert the training image data back to the original format of bounding boxes and dense masks based on the embedded tags.

[0086] In some embodiments, a dense mask matrix corresponding to each training image data is first obtained, where the dense mask matrix contains information about the category to which each pixel in the image belongs. The dense mask matrix is then divided into several small blocks to better capture local features.

[0087] A multi-layer perceptron is used to map these patches into a high-dimensional language embedding space, capturing the semantic information in the text. Through the above steps, each patch in the dense mask matrix corresponds to a fixed-length vector, allowing the model to better understand the local features in the image.

[0088] Step 304 : Supervise the bounding box predicted by the general vision model based on L1 loss and generalized intersection-over-union loss.

[0089] To ensure the model can accurately locate the target object, we use L1 loss to measure the difference between the predicted bounding box and the ground-truth bounding box, while gIoU loss is used to evaluate the degree of overlap between the two. These two metrics work together during the model training process to help the model better learn how to accurately locate the target object.

[0090] Step 305 : Calculate the sum of absolute differences between the predicted depth mask and the true depth mask using L1 loss, and optimize the general vision model based on the calculation result.

[0091] In order to improve the accuracy of the model in the semantic segmentation task, L1 loss is used to measure the difference between the predicted depth mask and the true depth mask, and the gradient is calculated through the backpropagation algorithm, and then the model parameters are updated to minimize the loss function.

[0092] Step 306: Determine the similarity between the predicted token and the next token in the real sentence based on the cross entropy loss, and optimize the visual general model according to the similarity.

[0093] The cross entropy loss is used to measure the difference between the predicted token and the true token, and the gradient is calculated through the backpropagation algorithm, and then the model parameters are updated to minimize the loss function.

[0094] In some embodiments, after the number of training times reaches a preset threshold, the training of the visual general model is stopped; or,

[0095] After the loss function is less than a preset loss function threshold, the training of the general vision model is stopped.

[0096] In practical applications, the conditions for stopping model training can be determined according to actual needs, and the embodiments of the present application do not limit this.

[0097] In some embodiments, after stopping training of the visual general model, the visual general model is used as a visual encoder of a multimodal large language model to perform visual understanding of the image to be recognized.

[0098] A general visual backbone network trained on object detection and pixel-level tasks is used as the visual encoder of a multimodal large language model in the visual instruction fine-tuning step. In some embodiments, the visual instruction fine-tuning stage can also be trained together with the large language model based on the already trained general visual model to achieve better visual-language alignment results and visual instruction following effects.

[0099] By uniformly training multiple visual task data through a language interface, a new universal visual backbone network is obtained. This network has better visual-language space alignment capabilities and can better process and capture visual information at various levels, effectively improving the capabilities and effectiveness of mainstream multimodal large language models. The specific steps of the present invention are as follows:

[0100] Construct question-answer data pairs based on various visual tasks. By constructing and acquiring a large amount of question-answer data through datasets such as visual question answering, object detection and recognition, and dense prediction, we can train general models that can be used for various visual tasks.

[0101] The general vision model is trained through a language interface. Based on the acquired training data, a language model that can process language tokens is used to generate answers based on the provided questions and visual inputs. The answers are then compared with the labels and the loss function is calculated to train the general vision model.

[0102] Combine the trained general model with the large language model. The trained general vision model can be used together with the existing large language model for visual command fine-tuning, replacing the traditional CLIP vision model, thereby achieving better vision-language alignment and visual command following.

[0103] Below we will introduce the key technologies of the present invention in detail.

[0104] Unifying vision tasks into a question-answering format.

[0105] In this paper, we design a solution to unify visual tasks into question-answering form. The question-answer pairs constructed in this paper are mainly divided into three aspects: vision-language tasks, object recognition tasks, and dense prediction tasks.

[0106] We introduce image description and visual question answering as vision-language tasks to align the visual and language modalities. The main goal of the vision-language task is to promote the alignment between visual and textual modalities, ensure that multimodal vision-general models can maintain their capabilities on vision-language tasks, and prevent them from overfitting to visual tasks.

[0107] To enhance the capabilities of our vision models, we built a series of object recognition tasks around the question-answering format. These tasks include object name prediction, quantity counting, region description, enumerating object locations, and answering questions within bounding boxes. To jointly train object recognition tasks with language data, we encode object detection bounding boxes into the language space using a lightweight multi-layer perceptron.

[0108] This paper enhances the model's ability to predict semantic, instance, and depth mask matrices from visual input by combining semantic segmentation, instance segmentation, and depth estimation tasks. We partition the dense mask matrix into a fixed number of blocks and align the dense mask matrix into the language embedding space using a multilayer perceptron. This discrete representation enables the language model to predict dense masks at a fine-grained level, promoting a direct correspondence between visual input and language output.

[0109] Leveraging natural language interfaces to enable training of general vision models.

[0110] Based on abundant question-answering training data from various vision tasks, this paper trains a universal vision model through a natural language interface. The model architecture consists of a universal vision encoder to be trained, a lightweight Transformer-based query generator (Q-former), and a language decoder that provides a language interface. We define a set of learnable vectors as image query vectors and interact with image features through the Q-former. These image query vectors are then concatenated with the language embedding vector for the language model to generate the next token. To enable joint training of multiple tasks, we mix all question-answer pairs during training, including vision-language, object recognition, and dense prediction. To efficiently and uniformly process bounding boxes, dense prediction masks, and depth masks, we use a set of lightweight encoders and decoders to handle specialized data in the language embedding space. After the decoding module, we convert the embedded tokens back to their original format of bounding boxes and dense masks, allowing for conventional output supervision. We use L1 and generalized intersection over union (gIoU) losses to supervise the predicted bounding boxes, binary cross entropy loss to supervise the segmentation masks, and L1 loss to supervise the depth masks. For the rest of the language tokens, we use cross-entropy loss to supervise the similarity between the predicted token and the next token in the true sentence.

[0111] Connecting a general vision model with an existing large language model.

[0112] This method designs a structured three-step process for training a vision-general model. In the initial stage, we train a basic alignment between the vision model and the language decoder. First, we fix the parameters of the visual backbone and train a Transformer-based Q-former, an image-query vector, and a language decoder with a low-rank adaptation module. This stage focuses primarily on vision-language tasks, encouraging the model to form connections between visual inputs and their language counterparts, thereby achieving initial cross-modal alignment. The second stage further expands the model's capabilities to include object detection and pixel-level vision tasks. In this stage, we unfreeze the visual backbone and begin joint training of the entire model. This holistic approach enables the vision-general model to refine its understanding of visual content in more detail, integrating object recognition and dense prediction tasks. In the final stage, we deprecate the Q-former and language interface decoder models, leveraging the vision-general backbone network trained on object detection and pixel-level tasks as the visual encoder for the visual instruction fine-tuning step of a multimodal large language model.

[0113] Corresponding to the above-mentioned image recognition model training method, the present invention also provides an image recognition model training device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, any details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment and will not be further described in this invention.

[0114] Figure 4 A structural diagram of a training device for an image recognition model provided in an embodiment of the present disclosure is shown in FIG. Figure 4 Shown, including:

[0115] An acquisition unit 41 is configured to acquire training question-answering data; wherein the training question-answering data is generated based on the training image data; and the training question-answering data includes questions and standard answers to the questions;

[0116] a training unit 42 for training a general vision model based on the training question-answer data and the training image data, to obtain a predicted answer to the training question-answer data generated by the general vision model;

[0117] The parameter adjustment unit 43 is used to calculate the loss function of the visual general model according to the standard answer and the predicted answer, and adjust the parameters of the visual general model according to the loss function.

[0118] The training device of the image recognition model provided by the present disclosure has the following main technical solutions: obtaining training question-answer data; wherein, the training question-answer data is generated based on training image data; the training question-answer data includes questions and standard answers to the questions; training a general visual model based on the training question-answer data and the training image data to obtain predicted answers to the training question-answer data generated by the general visual model; calculating the loss function of the general visual model based on the standard answers and the predicted answers, and adjusting the parameters of the general visual model based on the loss function. Compared with the related art, the embodiment of the present application unifies multiple tasks into question-answer data pairs through a solution, and uniformly trains multiple visual task data through a language interface, thereby obtaining a new general visual model. The network has better visual-language space alignment capabilities, and can better process and capture visual information at various levels, which can effectively improve the capabilities and effects of mainstream multimodal large language models.

[0119] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5 As shown, the training question-answering data includes visual-language conversion data, and the training unit 42 is further used to:

[0120] The training image data is input into the general vision model to obtain a predicted text description of the training image output by the general vision model.

[0121] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5As shown, the training question-answering data includes standard text descriptions of training image data, and the parameter adjustment unit 43 is further used to:

[0122] A loss function is calculated according to the predicted text description and the standard text description, and the visual general model is trained according to the loss function.

[0123] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5 As shown, the training question-answering data includes an object recognition task, and the training unit 42 includes:

[0124] A first training module 421 trains the general vision model based on object recognition tasks in the training question-and-answer data, wherein the object recognition tasks include at least one of object name prediction, quantity counting, region description, object location enumeration, and question-and-answering within a bounding box.

[0125] A second training module 422 is configured to generate a prediction token based on the concatenation of the image query vector and the language embedding vector in the general vision model;

[0126] The third training module 423 is used to obtain a dense mask matrix of the training image data, divide the dense mask matrix into a preset number of matrix blocks, align the dense mask matrix into a language embedding space through a multi-layer perceptron, and convert the training image data back to the original format of bounding boxes and dense masks based on the embedded tags.

[0127] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5 As shown, the parameter adjustment unit 43 is further used to:

[0128] Supervising the bounding box predicted by the general vision model based on L1 loss and generalized intersection-over-union loss;

[0129] Calculating the sum of absolute differences between the predicted depth mask and the true depth mask using L1 loss, and optimizing the general vision model based on the calculation result;

[0130] A similarity between the predicted token and a next token in the true sentence is determined based on a cross-entropy loss, and the visual general model is optimized according to the similarity.

[0131] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5 As shown, the device also includes:

[0132] a stop training unit 44 configured to stop training the visual general model after the parameter adjustment unit 33 calculates the loss function of the visual general model according to the standard answer and the predicted answer and adjusts the parameters of the visual general model according to the loss function, and after the number of training times reaches a preset number threshold; or

[0133] The training stopping unit 44 is further configured to stop training the general vision model after the loss function is less than a preset loss function threshold.

[0134] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5 As shown, the device also includes:

[0135] The visual understanding unit 45 is configured to use the visual general model as a visual encoder of a multimodal large language model to perform visual understanding of the image to be recognized after the stop training unit 44 stops training the visual general model.

[0136] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principles are the same, which is no longer limited in the embodiment of the present disclosure.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0138] Figure 6 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 6As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 502 or a computer program loaded from a storage unit 508 into a RAM (Random Access Memory) 503. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.

[0140] Various components in device 500 are connected to I / O interface 505, including: an input unit 506, such as a keyboard, mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, optical disk, etc.; and a communication unit 509, such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit 501 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various specialized AI (Artificial Intelligence) computing chips, various computing units that run machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the training method for the image recognition model. For example, in some embodiments, the training method for the image recognition model can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the aforementioned image recognition model training method in any other appropriate manner (for example, by means of firmware).

[0142] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System on Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0147] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0148] It's important to note that artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). This encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0149] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0150] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A training method for an image recognition model, characterized in that: include: Obtaining training question-answering data; wherein the training question-answering data is generated based on the training image data; the training question-answering data includes questions and standard answers to the questions; Training a general vision model based on the training question-answer data and the training image data to obtain a predicted answer to the training question-answer data generated by the general vision model; A loss function of the visual general model is calculated according to the standard answer and the predicted answer, and parameters of the visual general model are adjusted according to the loss function.

2. The method according to claim 1, characterized in that The training question-and-answer data includes visual-to-language conversion data, and the training of the general vision model based on the training question-and-answer data and the training image data to obtain a predicted answer to the training question-and-answer data generated by the general vision model further includes: The training image data is input into the general vision model to obtain a predicted text description of the training image output by the general vision model.

3. The method according to claim 2, characterized in that The training question-answer data includes standard text descriptions of training image data, and calculating the loss function of the general vision model based on the standard answers and the predicted answers, and adjusting parameters of the general vision model based on the loss function includes: A loss function is calculated according to the predicted text description and the standard text description, and the visual general model is trained according to the loss function.

4. The method according to claim 1, wherein The training question-and-answer data includes an object recognition task, and the training of a general vision model based on the training question-and-answer data and the training image data to obtain a predicted answer to the training question-and-answer data generated by the general vision model includes: Training the general vision model according to an object recognition task in the training question-and-answer data; wherein the object recognition task includes at least one of object name prediction, quantity counting, area description, listing target locations, and question-and-answering within a bounding box; Generate a prediction token based on the concatenation of the image query vector and the language embedding vector in the general vision model; Obtain a dense mask matrix of the training image data, divide the dense mask matrix into a preset number of matrix blocks, align the dense mask matrix into a language embedding space through a multilayer perceptron, and convert the training image data back to the original format of bounding boxes and dense masks based on the embedding labels.

5. The method according to claim 4, characterized in that Calculating the loss function of the visual general model according to the standard answer and the predicted answer, and adjusting parameters of the visual general model according to the loss function further includes: Supervising the bounding box predicted by the general vision model based on L1 loss and generalized intersection-over-union loss; Calculating the sum of absolute differences between the predicted depth mask and the true depth mask using L1 loss, and optimizing the general vision model based on the calculation result; A similarity between the predicted token and a next token in the true sentence is determined based on a cross-entropy loss, and the visual general model is optimized according to the similarity.

6. The method according to any one of claims 1 to 5, characterized in that After calculating the loss function of the visual general model according to the standard answer and the predicted answer, and adjusting parameters of the visual general model according to the loss function, the method further includes: After the number of training times reaches a preset number threshold, the training of the general vision model is stopped; or, after the loss function is less than a preset loss function threshold, the training of the general vision model is stopped.

7. The method according to claim 6, characterized in that After stopping the training of the general vision model, the method further includes: The visual general model is used as the visual encoder of a multimodal large language model to perform visual understanding of the image to be recognized.

8. A training device for an image recognition model, characterized in that: include: An acquisition unit, configured to acquire training question-answering data; wherein the training question-answering data is generated based on the training image data; and the training question-answering data includes questions and standard answers to the questions; a training unit, configured to train a general vision model based on the training question-answer data and the training image data, and obtain a predicted answer to the training question-answer data generated by the general vision model; A parameter adjustment unit is used to calculate the loss function of the visual general model based on the standard answer and the predicted answer, and to adjust the parameters of the visual general model based on the loss function.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images