Training method and device of multi-task visual model

Abstract

The invention provides a multi-task visual model training method and device, a multi-task visual model comprises a first visual encoder and a task decoder, and the method comprises the following steps: freezing the task decoder; obtaining training data; the training data comprises a sample image and a text pair associated with the sample image; and training the first visual encoder by using a language decoder in the trained first visual language model based on the sample image and text pair in the training data, thereby fully utilizing rich language features and semantic understanding ability learned by the language decoder, and improving the semantic understanding ability of the first visual encoder. According to the method, the first visual encoder can learn the associated information between the image and the text, so that the image feature extraction capability of the first visual encoder is improved, the first visual encoder can capture key information in the image more accurately, feature representation with higher quality is provided for a subsequent task decoder, and the user experience is improved. And the accuracy and robustness of the whole multi-task visual model on various tasks are improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method and apparatus for training a multi-task visual model. Background Technology

[0002] With the rapid development of artificial intelligence technology, visual processing technology has been widely applied in various fields such as computer vision, intelligent interaction, autonomous driving, medical image analysis, and industrial inspection. Multi-task visual models, as core components for realizing these applications, include visual encoders for extracting image features and task decoders for performing specific tasks (such as classification, detection, and segmentation). Effectively training the visual encoder has become crucial for improving the model's semantic understanding and generalization performance. Summary of the Invention

[0003] This disclosure provides a method and apparatus for training a multi-task visual model, to at least partially address one of the technical problems in related technologies. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a method for training a multi-task visual model is provided, the multi-task visual model including a first visual encoder and a task decoder, comprising: freezing the task decoder; acquiring training data; wherein the training data includes sample images and text pairs associated with the sample images; and training the first visual encoder based on the sample images and the text pairs in the training data using a language decoder in a trained first visual language model.

[0004] According to a second aspect of the present disclosure, a training apparatus for a multi-task visual model is provided, the multi-task visual model including a first visual encoder and a task decoder, comprising: a freezing module for freezing the task decoder; an acquisition module for acquiring training data; wherein the training data includes sample images and text pairs associated with the sample images; and a training module for training the first visual encoder based on the sample images and the text pairs in the training data using a language decoder in a trained first visual language model.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a training method for a multi-task visual model as described in the first aspect of the present disclosure.

[0006] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the training method of the multi-task visual model according to the first aspect of the embodiments of the present disclosure.

[0007] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises a computer program, which, when executed by a processor, implements the training method of the multi-task visual model according to the first aspect of the embodiments of the present disclosure.

[0008] The technical solutions provided by the embodiments of the present disclosure at least have the following beneficial effects: In the technical solution, by pre-freezing the task decoder during the training process, the complexity and uncertainty caused by the adjustment of the task decoder parameters during the training process are avoided, so that the focus of the training can be highly concentrated on the first visual encoder. Then, by training the first visual encoder using the trained language decoder in the first visual language model and combining the training data comprising the sample image and its associated text pair, the rich language features and semantic understanding ability learned by the language decoder can be fully utilized, so that the first visual encoder can learn more accurate and in-depth associated information between the image and the text. This targeted training method helps to improve the image feature extraction ability of the first visual encoder, so that the first visual encoder can more accurately capture the key information in the image, and then provide higher quality feature representation for the subsequent task decoder, which can significantly improve the accuracy and robustness of the multi-task visual model in various visual processing tasks. In addition, by obtaining the unannotated global image or the target entity subgraph of the global image as the sample image, combining the diversified outputs generated based on the prompt text by the plurality of trained second visual language models, further filtering high-quality texts and constructing text pairs, and finally forming a data set that can be used for model training according to the sample image and the text pair, the dependence on manual annotation of image content is completely eliminated, and the automatic generation of supervision signals is realized, which realizes the automatic construction of training data, greatly reduces the data preparation cost, and breaks through the dependence on manual annotation. In addition, when obtaining the target entity subgraph of the global image, the global image is subjected to target detection to obtain a plurality of detection boxes, and the target entity subgraph is determined according to the local image feature semantic similarity of the candidate entity subgraph in the detection box, which effectively removes the repeated and redundant information, avoids the waste of computing resources and the deviation of processing results caused by similar entity interference, and enables the model to focus on the key target entity, thereby improving the accuracy of target recognition.

[0009] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0011] Figure 1 is a flowchart of a training method of a multi-task visual model according to a first embodiment of the present disclosure; Figure 2 is a flowchart of a training method of a multi-task visual model according to a second embodiment of the present disclosure; Figure 3 is a flowchart of a training method of a multi-task visual model according to a third embodiment of the present disclosure; Figure 4 is a flowchart of a training method of a multi-task visual model according to a fourth embodiment of the present disclosure; Figure 5 is a schematic diagram of the principle of a training method of a multi-task visual model according to an embodiment of the present disclosure; Figure 6 is a structural diagram of a visual language model according to an embodiment of the present disclosure; Figure 7 is a structural diagram of a multi-task visual model according to an embodiment of the present disclosure; Figure 8 is a structural diagram of a training device of a multi-task visual model according to a fifth embodiment of the present disclosure; Figure 9 is a structural diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] In order to make the ordinary person in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0013] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0014] It should be noted that in the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the consent of the user and in accordance with relevant laws and regulations, and do not violate public order and good customs.

[0015] With the development of artificial intelligence technology, the visual model technology based on deep learning has made great progress and has been widely recognized in the industry. Through the way of visual encoder (also known as basic model) + model fine-tuning (Finetuning), it has been implemented in multiple scenarios. How to obtain a basic model with high representation ability and low inference cost is the key to this problem.

[0016] Early basic models are usually from classification pre-training of large-scale image recognition datasets (such as ImageNet). Through fine-grained classification and diverse data composition, the model can achieve better representation effect. With the expansion of landing scenarios, large-scale image recognition datasets have gradually reached a bottleneck, and the high data annotation cost cannot further improve the effect of the basic model.

[0017] In order to eliminate the cost of data annotation, a self-supervised learning basic model pre-training method appears. Through a large amount of unmarked data, the representation ability of the basic model is optimized through methods such as contrastive learning, mask restoration, and self-distillation. The model trained by self-supervised learning has better representation ability than classification pre-training and has achieved good performance in few-shot, zero-shot, and similar search tasks.

[0018] With the development of technology influenced and driven by large language model technology (LLM), vision-language model (Vision-Language Model, VLM for short) has received more attention. On the one hand, the accumulation of multi-modal datasets and the maturity of computing power have led to the emergence of large-scale vision-language models. In tasks such as image-text question answering, retrieval, and understanding, the performance has reached close to the average level of human beings, and the understanding of visual semantics has reached a new height. Some models have also launched image-text question answering functions.

[0019] In related technologies, the following methods are mainly used to train the basic model: (1) Mask restoration self-supervised method: imitating the pre-training method of BERT, the image is divided into small patches of the same size, and a part of it is masked with uniform pixel values in a random manner. The feature values of the masked part are predicted by the unmasked image patches, and the basic model is optimized by mean square error or cosine similarity loss; (2) Self-distillation base model training: a teacher model and a student model are constructed during the training process, the two model structures are completely consistent, the parameters of the teacher model are updated through the Exponential Moving Average (EMA) method, and the student model is updated through error back propagation. During training, the input samples are transformed by cropping, rotating, normalizing, etc. The samples before and after transformation are input into the teacher model and the student model respectively, and the loss function value is constructed by aligning the two model output feature values. Based on the loss function value, the base model is trained; (3) Weakly supervised algorithm: a double tower model is constructed, one side is a text encoder (Text Encoder) and the other side is a visual encoder (Image Encoder). During training, input the image-text pair, that is, an image and a text description of the image content. The loss function value is constructed by aligning the image-text semantics, and the base model is trained based on the loss function value; (4) Train VLM: a large amount of image-text question and answer pair data is needed during training, that is, one or more images and one or more image-text question and answer content. VLM is divided into three basic parts: visual encoder, large language model, and visual semantic adapter. During the training process, different data is used for multi-stage training, and different model parameters are systematically optimized to optimize the performance of the visual encoder. The above-mentioned self-supervised method of mask restoration has the lowest data labeling cost, and can be trained as long as a large enough data scale is available. However, due to the lack of text semantic information in the training process, the semantic representation ability of the model is weak, and more data is needed when applying the model training. For the weakly supervised algorithm, a large amount of image-text pair data is needed, which has rich data reserves in the Internet scenario. However, in the financial, industrial and manufacturing scenarios, due to data security issues, it is impossible to realize the large-scale output of such data, so it is difficult to apply this method in the above-mentioned scenarios. For VLM, existing open source multi-modal large models have good semantic question and answer effect, but due to the large size of the visual model, it cannot be directly connected to the low-cost and high-concurrency visual analysis scene.

[0020] To solve any of the above problems, the present disclosure provides a multi-task visual model training method and device.

[0021] The multi-task visual model training method and device of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0022] Figure 1 is a flowchart of the multi-task visual model training method according to the first embodiment of the present disclosure.

[0023] As shown in Figure 1 The training method of the multi-task visual model includes the following steps: Step 110, freezing the task decoder.

[0024] The multi-task visual model includes a first visual encoder and a task decoder.

[0025] In the embodiments of the present disclosure, the multi-task visual model includes a first visual encoder and a task decoder. In order to avoid the interference of the parameter change of the task decoder on the feature learning process, as a possible implementation manner, the task decoder in the multi-task visual model can be frozen, that is, the structure and parameters of the task decoder are kept unchanged, and the first visual encoder is independently trained and optimized.

[0026] Step 120, obtaining training data; wherein the training data includes a sample image and a text pair associated with the sample image.

[0027] In order to effectively train the first visual encoder and improve the feature extraction capability of the first visual encoder, as a possible implementation manner, training data containing a sample image and a text pair associated with the sample image can be obtained. The sample image refers to original image data used for model input, which can include visual content associated with a second target processing task (visual language processing task). The text pair refers to a set of literal information related to the semantics of the sample image. The text pair can include prompt text, which is used to prompt the first visual language model to perform a first target processing task. The text pair also includes reference output text associated with the first target processing task. The second target processing task and the first target processing task can be the same visual language processing task or different visual language processing tasks, which are not limited in the present disclosure.

[0028] In some embodiments, as shown in Figure 2 Step 120 can further include the following steps: Step 1201, obtaining a sample image; wherein the sample image includes a global image associated with the second target processing task or a target entity sub-image in the global image. The global image does not contain annotation information associated with the image task.

[0029] To construct training data suitable for a specific visual processing task, one possible approach is to acquire sample images. These sample images include a global image related to a second target processing task or a target entity sub-image cropped from the global image. The second target processing task can be a visual language processing task to be performed by the visual language model. The global image can be, for example, an entire scene image, and the target entity sub-image can be, for example, a local region of an object in the global image. Neither the global image nor the target entity sub-image contains manually labeled information related to the target task, such as category labels, bounding boxes, or segmentation masks.

[0030] In some embodiments, such as Figure 3 As shown, step 1201 may specifically include the following steps: Step 12011: Obtain the global image associated with the second target processing task.

[0031] As one possible implementation, raw images related to a second objective processing task are collected as global images, where the second objective processing task may include: image description generation, visual question answering, visual reasoning, etc. Step 12012: Perform target detection on the global image to obtain multiple detection boxes in the global image.

[0032] To achieve the decomposition from the global image to local regions, one possible approach is to automatically identify potentially interesting object regions in the global image using an object detection model and generate multiple detection boxes.

[0033] Step 12013: Determine the target entity sub-image from the candidate entity sub-images within each detection box based on the semantic similarity between the local image features of the candidate entity sub-images within each detection box in the global image.

[0034] To reduce interference from irrelevant regions, one possible approach is to calculate the semantic similarity between local image features of candidate entity sub-images within each detection box, and then select representative or most frequently occurring entity regions as target entity sub-images.

[0035] As an example, semantic features are extracted from candidate entity sub-images within each detection box in the global image to obtain local image features of candidate entity sub-images within each detection box; based on the semantic similarity between the local image features of candidate entity sub-images within each detection box, duplicate candidate entity sub-images within each detection box are deduplicated to obtain retained candidate entity sub-images; based on the retained candidate entity sub-images, the target entity sub-image is determined.

[0036] In other words, semantic feature extraction is performed on the candidate entity sub-images within each detection box in the global image to obtain the local image features corresponding to the candidate entity sub-images within each detection box. Then, based on the semantic similarity between these local image features, deduplication is performed on the candidate entity sub-images within each detection box. For example, if two or more candidate entity sub-images have high semantic similarity, it means that they are relatively close at the semantic level and there may be duplication. In this case, the duplicate sub-images will be removed, and finally, the retained candidate entity sub-images will be obtained. Then, these retained candidate entity sub-images are used as the target entity sub-images.

[0037] Step 12014: Select sample images from the global image and the target entity sub-image according to the preset sample distribution ratio.

[0038] It is important to understand that since the global image provides rich scene context information, it helps the model learn the semantic features of images in different scenes; while the target entity sub-image focuses on specific entities, it allows the model to better learn the detailed features of entities. Therefore, in order to improve the generalization ability of the model, a certain number of images are selected as samples from the global image and the target entity sub-image according to the set sample distribution ratio.

[0039] For example, if the sample distribution ratio is 60% global images and 40% target entity sub-images, when selecting samples, the corresponding number of images are extracted from the global image set and the target entity sub-image set respectively according to this ratio, and combined into the final sample image set.

[0040] In summary, by acquiring a global image associated with a visual processing task adapted to the multi-task visual model, object detection is performed on the global image to obtain multiple detection boxes. The target entity sub-image is determined based on the semantic similarity of local image features of candidate entity sub-images within the detection boxes. This effectively removes redundant and repetitive information, avoids wasted computational resources and processing result deviations caused by interference from similar entities, and allows the model to focus on key target entities, improving the accuracy of target recognition. Finally, sample images are selected from the global image and target entity sub-images according to a preset sample distribution ratio, ensuring the diversity and representativeness of the samples. This allows the model to learn the overall features of the scene and deeply grasp the detailed features of the target entities, thereby enhancing the model's adaptability and generalization performance to different types of images. This improves the processing effect and reliability of the multi-task visual model in various practical application scenarios.

[0041] Step 1202: Invoke multiple trained second visual language models to process the sample images based on the prompt text, so as to obtain the output text of multiple second visual language models.

[0042] It should be understood that each trained second visual language model has strong reasoning and language generation capabilities on large-scale image-text data. Therefore, in order to achieve multi-angle understanding of the semantic content of the sample image and high-quality text generation, as a possible implementation manner, multiple trained second visual language models are used to generate multiple independent output texts for the same sample image and the same prompt text; wherein the prompt text is, for example, "please describe this picture" or "what objects are in the picture?".

[0043] In step 1203, a text pair associated with the sample image is generated according to the output texts of the multiple second visual language models and the prompt text.

[0044] In order to construct high-quality training data, as a possible implementation manner, the output texts of the second visual language models are quality evaluated, high-quality output texts are screened, and a text pair associated with the sample image is generated in combination with the prompt text.

[0045] As an example, the output texts of the second visual language models are quality evaluated to obtain quality scores of the output texts; and a text pair associated with the sample image is generated according to the output texts with quality scores higher than a set score threshold and the prompt text.

[0046] In order to effectively improve the accuracy and reliability of automatically generated training data, as a possible implementation manner, the output texts generated by each second visual language model are scored by using indicators such as semantic integrity, keyword coverage, logical consistency, or relevance to the prompt text to obtain corresponding quality scores; then, only high-quality texts with scores exceeding a preset threshold are retained, and fuzzy, incorrect, or redundant generated results are filtered out; finally, these high-quality texts are used as reference outputs to generate text pairs with corresponding prompt texts.

[0047] In order to improve the accuracy of quality evaluation of the output texts of the second visual language models, the quality evaluation of the output texts of the second visual language models can be performed by the following steps: (1) For any output text of a second visual language, the quality of the output text of the second visual language is evaluated according to multiple set evaluation dimensions to obtain quality sub-scores of the output text of the second visual language in each set evaluation dimension; In the embodiments of the present disclosure, for the output text of any second visual language model, a plurality of set evaluation dimensions are determined, wherein the plurality of dimensions refer to indicators for measuring the quality of the output text from different angles, for example, an accuracy dimension for indicating the degree of correspondence between the information conveyed by the output text and the actual situation, a fluency dimension for indicating whether the text sentence is smooth, whether there are obvious grammatical errors or harsh expressions, a richness dimension for judging whether the text content is detailed and the information amount is sufficient, and a relevance dimension for determining the closeness of the output text to the given visual input or task requirements, and then based on these set evaluation dimensions, the output text of the second visual language model is analyzed, and according to the specific standards and requirements of each dimension, the quality sub-scores of the output text on each dimension are respectively given. For example, in the accuracy dimension, if the output text is highly consistent with the true information, a higher score is given, and if there is deviation or error, the score is deducted according to the degree of deviation.

[0048] (2) The weights of each set evaluation dimension are weighted and summed to obtain the quality score of the output text of any second visual language model. Since the importance of different evaluation dimensions for measuring the overall quality of the output text is different, for example, in some scenarios where the accuracy of information is required, the importance of the accuracy dimension may be higher than that of the fluency dimension, therefore, the corresponding weights are set for each set evaluation dimension, and these weights reflect the relative importance of each dimension in the overall quality evaluation, and then by multiplying the quality sub-score of each dimension by the corresponding weight, and summing all the weighted sub-scores, the quality score of the output text of the second visual language model can be obtained.

[0049] For example, if the accuracy dimension weight is 0.5, the corresponding quality sub-score is 8, the fluency dimension weight is 0.3, the quality sub-score is 7, and the richness dimension weight is 0.2, the quality sub-score is 6, then the quality score = 8x0.5+7x0.3+6x0.2 = 4+2.1+1.2 = 7.3.

[0050] Step 1204, generating training data according to the sample image and the text pair.

[0051] Then, the text pair is paired with the corresponding sample image to generate the training data.

[0052] In summary, by obtaining a globally unannotated image or a target entity subgraph of the globally unannotated image as a sample image, combining a plurality of trained second visual language models to generate diversified outputs based on prompt texts, further screening high-quality texts and constructing text pairs, and finally forming a dataset that can be used for model training according to the sample image and the text pair, the need for manual annotation of image content is eliminated, and the generation of supervision signals is completely dependent on large models, realizing the automatic construction of training data, greatly reducing the data preparation cost, and breaking the dependence on manual annotation.

[0053] Step 130, using the language decoder in the trained first visual language model, based on the sample image and the text pair in the training data, training the first visual encoder.

[0054] In order to realize the effective training of the first visual encoder, as a possible implementation manner, the language decoder in the trained first visual language model is used to construct a cross-modal supervision signal based on the sample image and the text pair in the training data, and the first visual encoder is trained.

[0055] Among them, the first visual language model further includes a second visual encoder, and the parameter amount of the first visual encoder is less than the parameter amount of the second visual encoder; the parameter amount of the task decoder is less than the parameter amount of the language decoder, so that the multi-task visual model is more lightweight as a whole, faster in inference speed, and lower in calculation resource consumption.

[0056] In some embodiments, as shown in Figure 4 Step 130 can include the following steps: Step 1301, inputting the sample image into the first visual encoder to obtain the image semantic features of the sample image.

[0057] In order to realize the extraction of visual representation with semantic meaning from the sample image, as a possible implementation manner, the sample image is input into the first visual encoder to obtain the image semantic features of the sample image, wherein the first visual encoder performs block, embedding and self-attention modeling on the sample image, and converts the original pixel data of the sample image into semantic features in the form of high-dimensional vectors, that is, the image semantic features of the sample image output by the first visual encoder.

[0058] Step 1302, using the language decoder of the trained first visual language model to perform a target processing task on the image semantic features based on the prompt text to obtain the predicted output text.

[0059] In order to realize the semantic driven processing of image semantic features, as a possible implementation manner, the language decoder of the trained first visual language model is adopted, the language knowledge and task understanding ability learned by the language decoder in the large-scale image-text data are used to prompt the text as a guide to decode the image semantic features, and the natural language output conforming to the semantic logic is generated, that is, the predicted output text is obtained.

[0060] In step 1303, the loss function value is determined according to the difference between the predicted output text and the reference output text.

[0061] In order to accurately optimize the model parameters, as a possible implementation manner, the inconsistency between the predicted output text and the reference output text in the semantic or lexical level is calculated, and the inconsistency is quantified as a scalar loss value, for example, the loss value is calculated by adopting the cross-entropy loss, the BLEU score combined loss or the semantic similarity loss, etc. It should be noted that the loss function value reflects the deviation degree between the current output of the model and the expected output, which is used as a training signal for the backward propagation to guide the model parameter update.

[0062] In step 1304, the first visual encoder is trained according to the loss function value.

[0063] In order to realize the optimization of the image feature extraction capability, as a possible implementation manner, the gradient corresponding to the loss function value is transmitted to the first visual encoder by the back propagation algorithm, and the network parameters of the first visual encoder are updated, so that the first visual encoder generates image semantic features more conforming to the task requirements in the subsequent forward calculation.

[0064] As an example, the semantic features of the predicted output text are extracted to obtain the first semantic features of the predicted output text, the semantic features of the reference output text are extracted to obtain the second semantic features of the reference output text, and the loss function value is determined according to the difference between the similarity of the first semantic features and the second semantic features and the target similarity.

[0065] That is, in order to train the first visual encoder with more accurate semantics, the semantic features of the predicted output text are extracted to obtain the first semantic features, and the semantic features of the reference output text are extracted to obtain the second semantic features, the semantic similarity (such as cosine similarity) between the first semantic features and the second semantic features is calculated, and the calculated similarity is compared with the target similarity to obtain the difference between the similarity and the target similarity, and the loss function value is constructed according to the difference, and then the first visual encoder is trained according to the loss function value.

[0066] The loss function value is positively correlated with the above-mentioned differences; that is, the smaller the difference, the smaller the loss function value, and vice versa. Thus, in this disclosure, the first visual encoder can be trained based on the loss function value to minimize the loss function value.

[0067] It should be noted that the above example only uses minimizing the loss function value as the termination condition for training the first visual encoder. In actual applications, other termination conditions can also be set, such as the number of training iterations reaching a set threshold, the training duration exceeding a set duration threshold, etc. This disclosure does not impose any restrictions on these conditions.

[0068] In summary, by inputting sample images into the first visual encoder to extract semantic features, and using the language decoder of the trained first visual language model in combination with the prompt text to perform task-driven decoding of the image features, predictive output text is generated. Then, the loss function value is calculated based on the difference between the predicted text and the reference text, thereby optimizing the first visual encoder in reverse. Thus, the rich cross-modal semantic knowledge of the large model is fully utilized as a supervision signal, and the first visual encoder is effectively trained without the need for manual annotation of complex labels.

[0069] The training method for the multi-task visual model in this disclosure avoids the complexity and uncertainty caused by adjusting the parameters of the task decoder during training by pre-freezing the task decoder during the training process. This allows the training focus to be highly concentrated on the first visual encoder. Then, by using the language decoder in the trained first visual language model, combined with training data containing sample images and their associated text pairs, the first visual encoder is trained. This fully utilizes the rich language features and semantic understanding capabilities already learned by the language decoder, enabling the first visual encoder to learn the association information between images and text. This targeted training method helps improve the first visual encoder's ability to extract image features, allowing it to more accurately capture key information in images. This provides higher-quality feature representations for subsequent task decoders, significantly improving the accuracy and robustness of the multi-task visual model in various visual processing tasks.

[0070] Based on any of the above embodiments, such as Figure 5 As shown, the training method for the multi-task vision model in this embodiment may include the following steps: Step 510, Vertical Industry Unlabeled Dataset First, prepare an unlabeled dataset for the vertical industry. After a pre-labeling process for training data, a training dataset for model training is formed. The specific steps are as follows: Step 5101, Image data preprocessing By introducing a detection model trained based on a data set containing multiple objects (such as yolov5-obj365), the position of the entity subgraph contained in the global image is obtained, and according to the detection box information, the subgraph is intercepted. By comparing the language-image pre-training (CLIP) feature comparison method, data deduplication is performed, and according to the sample distribution, the global graph and subgraph ratio is controlled at an appropriate ratio (such as 1:5). In this way, the output visual model has good representation ability for global and local images.

[0071] Step 5102, based on the training set of open source VLM The prepared data set is sequentially called to the open source VLM model, and a prompt text is randomly selected, such as "describe the entity in the image and the relationship between the entity", "explain the main entity in the image and the relationship between the secondary entity and it", "describe the semantic content in the image as much as possible. Granularity description" and the like.

[0072] 5103, data set sampling and screening For the output data set, sampling inspection is performed. Since the open source VLM model is trained based on different data sets, some models may have low accuracy for the description content in the current vertical industry. Through the sampling accuracy statistical method, the results of some models can be excluded from the training set.

[0073] The training set used for model training is obtained through the above steps, and the training set is composed of images and question and answer pairs. Each image will have one or more text question and answer contents. For each image I from the image set That is , there is a question and answer pair set , which together constitute the training data set: .

[0074] Step 520, model training As shown in Figure 6 , the VLM model is divided into two parts, a vision encoder (Vision Transformer, ViT) structure, , input image , to obtain image semantic features , the relevant parameters are ; and a large language model (LLM), , input image semantic features and related questions , output description , the relevant parameters are .

[0075] In model training, first, an open source VLM model is selected to obtain its LLM model parameters , a visual encoder is newly created for the computing power limit of the online model , the parameters are , since does not require strong generality, the parameter amount is usually an order of magnitude smaller than , and the loss function value is determined by cross entropy, and the specific formula is as follows: Loss ; The model training goal is to optimize the parameters to minimize the loss function value, that is . The visual encoder obtained in this way can have strong visual representation ability under the vertical industry video input domain.

[0076] Further, as shown in Figure 7 , by accessing different algorithm task decoders, downstream application algorithm models can be quickly developed. Since the ViT algorithm framework is used, the basic model can be adjusted by lightweight model fine-tuning techniques such as Adapter and LoRa.

[0077] Corresponding to the training method of the multi-task visual model provided in the above embodiment, the present disclosure also provides a training device of a multi-task visual model. Since the training device of the multi-task visual model provided in the embodiment of the present disclosure corresponds to the training method of the multi-task visual model provided in the above embodiment, the business processing method embodiment is also applicable to the training device of the multi-task visual model provided in the embodiment of the present disclosure. In the embodiment of the present disclosure, it will not be described in detail.

[0078] Figure 8 is a structural schematic diagram of the training device of the multi-task visual model shown in the fifth embodiment of the present disclosure. The multi-task visual model includes a first visual encoder and a task decoder.

[0079] As shown in Figure 8 , the training device 800 of the multi-task visual model includes a freezing module 810, an acquisition module 820, and a training module 830.

[0080] The freezing module 810 is configured to freeze the task decoder, the obtaining module 820 is configured to obtain training data, and the training data includes a sample image and a text pair associated with the sample image, and the training module 830 is configured to train the first visual encoder based on the sample image and the text pair in the training data by using the language decoder in the trained first visual language model.

[0081] As a possible implementation manner, the text pair includes prompt text, and the prompt text is used to prompt a first target processing task to be performed by the first visual language model, and the text pair further includes reference output text associated with the first target processing task, the training module 830 is configured to input the sample image into the first visual encoder to obtain image semantic features of the sample image, perform the first target processing task on the image semantic features based on the prompt text by using the language decoder in the trained first visual language model to obtain predicted output text, and determine a loss function value according to a difference between the predicted output text and the reference output text, and train the first visual encoder according to the loss function value.

[0082] As a possible implementation manner, the training module 830 is configured to perform semantic feature extraction on the predicted output text to obtain first semantic features of the predicted output text, perform semantic feature extraction on the reference output text to obtain second semantic features of the reference output text, and determine the loss function value according to a difference between a similarity between the first semantic features and the second semantic features and a target similarity.

[0083] As a possible implementation manner, the obtaining module 820 is configured to obtain the sample image, and the sample image includes a global image associated with a second target processing task or a target entity subgraph in the global image, and the global image does not contain annotation information associated with the image task, call a plurality of trained second visual language models, perform the second target processing task on the sample image based on the prompt text to obtain output text of the plurality of second visual language models, generate the text pair associated with the sample image according to the output text of the plurality of second visual language models and the prompt text, and generate the training data according to the sample image and the text pair.

[0084] As a possible implementation manner, the obtaining module 820 is configured to perform quality evaluation on the output text of each second visual language model to obtain a quality score of each output text, and generate the text pair associated with the sample image according to the output text with the quality score higher than a set score threshold and the prompt text.

[0085] As a possible implementation manner, the acquisition module 820 is configured to acquire a global image associated with a second target processing task adapted to the multi-task visual model; perform target detection on the global image to obtain a plurality of detection boxes in the global image; determine a target entity subgraph from candidate entity subgraphs in each detection box in the global image according to semantic similarity between local image features of the candidate entity subgraphs in each detection box; and select a sample image from the global image and the target entity subgraph according to a preset sample distribution ratio.

[0086] As a possible implementation manner, the acquisition module 820 is configured to perform semantic feature extraction on the candidate entity subgraphs in each detection box in the global image to obtain local image features of the candidate entity subgraphs in each detection box; and perform deduplication processing on the candidate entity subgraphs in each detection box according to semantic similarity between the local image features of the candidate entity subgraphs in each detection box to obtain retained candidate entity subgraphs; and determine the target entity subgraph according to the retained candidate entity subgraphs.

[0087] As a possible implementation manner, the acquisition module 820 is configured to, for the output text of any second visual language model, perform quality evaluation on the output text of any second visual language model according to a plurality of set evaluation dimensions to obtain quality sub-scores of the output text of any second visual language model in each set evaluation dimension; and perform weighted summation on the quality sub-scores of the output text of any second visual language model according to weights of each set evaluation dimension to obtain a quality score of the output text of any second visual language model.

[0088] As a possible implementation manner, the first visual language model further comprises a second visual encoder, a parameter quantity of the first visual encoder is less than a parameter quantity of the second visual encoder; and a parameter quantity of the task decoder is less than a parameter quantity of the language decoder. The training device of the multi-task visual model in the embodiment of the present disclosure avoids the complexity and uncertainty caused by the adjustment of the parameters of the task decoder during the training process by pre-freezing the task decoder during the training process, so that the focus of the training can be highly concentrated on the first visual encoder. Furthermore, the first visual encoder is trained by using the language decoder in the trained first visual language model in combination with the training data comprising the sample image and the associated text pair, which can fully utilize the rich language features and semantic understanding ability learned by the language decoder, so that the first visual encoder can learn more accurate and in-depth association information between images and texts. This targeted training method helps to improve the image feature extraction capability of the first visual encoder, so that the first visual encoder can more accurately capture the key information in the image, thereby providing higher quality feature representation for the subsequent task decoder, which can significantly improve the accuracy and robustness of the multi-task visual model in various visual processing tasks.

[0089] In an example embodiment, an electronic device is also presented.

[0090] The electronic device comprises: a processor; a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the training method of the multi-task visual model as presented in any of the preceding embodiments.

[0091] As an example, Figure 9 is a structural schematic diagram of an electronic device 900 according to an example embodiment of the present disclosure, as Figure 9 The electronic device 900 can further comprise: a memory 910 and a processor 920, a bus 930 connecting different components (including the memory 910 and the processor 920), the memory 910 storing a computer program, and the processor 920 implementing the training method of the multi-task visual model according to the embodiments of the present disclosure when executing the program.

[0092] The bus 930 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0093] The electronic device 900 typically includes a variety of electronic device readable media. These media can be any available media that is accessible by the electronic device 900 and includes both volatile and non-volatile media, removable and non-removable media.

[0094] The memory 910 can also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 940 and / or cache memory 950. The electronic device 900 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 960 can be used for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 9 (not shown), commonly referred to as a "hard disk drive"). Although Figure 9A disk drive, a floppy disk drive, and / or other memory unit 910, can be provided, as is well known to those skilled in the art. These and other removable storage units 910 can be read from and / or written to by a removable storage interface 912. The removable storage interface 912 can be a program / utility 980 that will include appropriate driver software as is well known in the art for a mechanical interface to the removable storage unit 910. The program / utility 980 contains the necessary processes to load, store, and / or update the program modules 970.

[0095] Program / utility 980 having a set (e.g., at least one) of program modules 970 can be stored in memory 910 and implemented by the processor 920. The program modules 970 can include, but are not limited to, an operating system, one or more applications, other program modules, and program data, each of which can include implementations of the embodiments of the disclosure described herein, or some combination thereof. The program modules 970 are generally executed by the processor 920 to implement the processes and methods of the embodiments of the disclosure described herein.

[0096] The electronic device 900 can also communicate with one or more external devices 990 such as a keyboard or a pointing device, displays 991, etc.; other devices such as a storage device or an external effects device (not shown); and / or one or more devices that enable a user to interact with the electronic device 900; and / or one or more devices that enable the electronic device 900 to communicate with one or more other computing devices. Such communication can be via an input / output (I / O) interface 992. The electronic device 900 can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the public network such as the Internet) through a network adapter 993. As depicted, the network adapter 993 is in communication with the other modules of the electronic device 900 through the bus 930. It should be appreciated that other hardware and / or software modules can be used in conjunction with the electronic device 900, including but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0097] The processor 920 performs functions of various applications and data processing by running programs stored in the memory 910.

[0098] It should be noted that the implementation process and technical principles of the electronic device of the embodiment are described above in the explanation of the training method of the multi-task visual model of the embodiment of the disclosure, which will not be described here.

[0099] In an example embodiment, a computer readable storage medium, such as a memory including instructions, is also provided, which can be executed by a processor of an electronic device to complete the training method of the multi-task visual model proposed in any of the above embodiments. Optionally, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0100] In an example embodiment, a computer program product is also provided, which includes computer programs / instructions, characterized in that the computer programs / instructions are executed by a processor to implement the training method of the multi-task visual model proposed in any of the above embodiments.

[0101] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known use or custom in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0102] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A training method for a multi-task visual model, characterized in that, The multi-task vision model includes a first visual encoder and a task decoder, including: Freeze the task decoder; Acquire training data; wherein the training data includes sample images and text pairs associated with the sample images; The first visual encoder is trained using the language decoder in the trained first visual language model, based on the sample images and text pairs in the training data.

2. The method according to claim 1, characterized in that, The text pair includes prompt text, wherein The prompt text is used to prompt the first visual language model to perform a first target processing task, and the text pair also includes: reference output text associated with the first target processing task; The step of training the first visual encoder using the language decoder in the trained first visual language model, based on the sample images and text pairs in the training data, includes: The sample image is input into the first visual encoder to obtain the image semantic features of the sample image; The language decoder of the first visual language model trained thereon performs the first target processing task on the semantic features of the image based on the prompt text, and obtains the predicted output text. The loss function value is determined based on the difference between the predicted output text and the reference output text; The first visual encoder is trained based on the loss function value.

3. The method according to claim 2, characterized in that, The step of determining the loss function value based on the difference between the predicted output text and the reference output text includes: Semantic features are extracted from the predicted output text to obtain the first semantic features of the predicted output text; Semantic features are extracted from the reference output text to obtain the second semantic features of the reference output text; The loss function value is determined based on the difference between the similarity between the first semantic feature and the second semantic feature and the target similarity.

4. The method according to claim 1, characterized in that, The acquisition of training data includes: Obtain sample images; wherein, the sample images include a global image associated with the second target processing task or a target entity sub-image in the global image, and the global image does not contain annotation information associated with the target processing task; Multiple trained second visual language models are invoked to perform a second target processing task on the sample image based on the prompt text, so as to obtain the output text of the multiple second visual language models; Based on the output text of the multiple second visual language models and the prompt text, generate text pairs associated with the sample image; The training data is generated based on the sample images and the text pairs.

5. The method according to claim 4, characterized in that, The step of generating text pairs associated with the sample image based on the output text of the plurality of second visual language models and the prompt text includes: The output text of each of the second visual language models is evaluated for quality to obtain a quality score for each output text; Based on the output text whose quality score is higher than a set score threshold and the prompt text, a text pair associated with the sample image is generated.

6. The method according to claim 4, characterized in that, The acquisition of sample images includes: Obtain the global image associated with the second target processing task; Target detection is performed on the global image to obtain multiple detection boxes in the global image; Based on the semantic similarity between the local image features of the candidate entity sub-images within each detection box in the global image, the target entity sub-image is determined from the candidate entity sub-images within each detection box; The sample image is selected from the global image and the target entity sub-image according to the preset sample distribution ratio.

7. The method according to claim 6, characterized in that, The step of determining the target entity sub-image from the candidate entity sub-images within each detection box based on the semantic similarity between the local image features of each candidate entity sub-image within each detection box in the global image includes: Semantic features are extracted from the candidate entity sub-images within each detection box in the global image to obtain the local image features of the candidate entity sub-images within each detection box. Based on the semantic similarity between the local image features of the candidate entity sub-images within each detection box, the candidate entity sub-images within each detection box are deduplicated to obtain the retained candidate entity sub-images. The target entity subgraph is determined based on the retained candidate entity subgraphs.

8. The method according to claim 5, characterized in that, The quality evaluation of the output text of each of the second visual language models to obtain a quality score for each output text includes: For the output text of any second visual language model, the quality of the output text of the second visual language model is evaluated according to multiple set evaluation dimensions to obtain the quality sub-scores of the output text of the second visual language model in each set evaluation dimension; The quality sub-scores of the output text of any second visual language model are weighted and summed according to the weights of each of the defined evaluation dimensions to obtain the quality score of the output text of any second visual language model.

9. The method according to claim 1, characterized in that, The first visual language model further includes a second visual encoder, the number of parameters of the first visual encoder being less than the number of parameters of the second visual encoder; the number of parameters of the task decoder is less than the number of parameters of the language decoder.

10. A training device for a multi-task visual model, characterized in that, The multi-task vision model includes a first visual encoder and a task decoder, including: A freeze module is used to freeze the task decoder; An acquisition module is used to acquire training data; wherein the training data includes sample images and text pairs associated with the sample images; The training module is used to train the first visual encoder based on the sample images and text pairs in the training data, using the language decoder in the trained first visual language model.

Images