Method and apparatus for training multi-task visual model

Abstract

The present disclosure provides a method and device for training a multi-task visual model, the multi-task visual model comprising a first visual encoder and a task decoder, the method comprising: freezing the task decoder; obtaining training data; the training data comprising sample images and text pairs associated with the sample images; training the first visual encoder based on the sample images and the text pairs in the training data using a language decoder in the trained first visual language model, whereby the rich language features and semantic understanding capabilities learned by the language decoder can be fully utilized, the first visual encoder can learn the association information between images and texts, the extraction capability of the first visual encoder for image features is improved, the first visual encoder can more accurately capture the key information in the images, provide higher quality feature representation for the subsequent task decoder, and improve the accuracy and robustness of the entire multi-task visual model on various tasks.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method and apparatus for training a multi-task visual model. Background Technology

[0002] With the rapid development of artificial intelligence technology, visual processing technology has been widely applied in various fields such as computer vision, intelligent interaction, autonomous driving, medical image analysis, and industrial inspection. Multi-task visual models, as core components for realizing these applications, include visual encoders for extracting image features and task decoders for performing specific tasks (such as classification, detection, and segmentation). Effectively training the visual encoder has become crucial for improving the model's semantic understanding and generalization performance. Summary of the Invention

[0003] This disclosure provides a method and apparatus for training a multi-task visual model, to at least partially address one of the technical problems in related technologies. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a method for training a multi-task visual model is provided, the multi-task visual model including a first visual encoder and a task decoder, comprising: freezing the task decoder; acquiring training data; wherein the training data includes sample images and text pairs associated with the sample images; and training the first visual encoder based on the sample images and the text pairs in the training data using a language decoder in a trained first visual language model.

[0004] According to a second aspect of the present disclosure, a training apparatus for a multi-task visual model is provided, the multi-task visual model including a first visual encoder and a task decoder, comprising: a freezing module for freezing the task decoder; an acquisition module for acquiring training data; wherein the training data includes sample images and text pairs associated with the sample images; and a training module for training the first visual encoder based on the sample images and the text pairs in the training data using a language decoder in a trained first visual language model.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a training method for a multi-task visual model as described in the first aspect of the present disclosure.

[0006] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a training method for a multi-task visual model as described in the first aspect of the present disclosure.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the training method for a multi-task visual model as described in the first aspect of the present disclosure.

[0008] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: In this technical solution, by pre-freezing the task decoder during training, the complexity and uncertainty caused by adjusting the task decoder parameters during training are avoided. This allows the training focus to be highly concentrated on the first visual encoder. Then, by using the language decoder in the trained first visual language model, combined with training data containing sample images and their associated text pairs, the first visual encoder is trained. This fully leverages the rich linguistic features and semantic understanding capabilities already learned by the language decoder, enabling the first visual encoder to learn more accurate and in-depth associations between images and text. This targeted training method helps improve the first visual encoder's ability to extract image features, allowing it to more accurately capture key information in images. This provides higher-quality feature representations for subsequent task decoders, significantly improving the accuracy and robustness of the multi-task visual model across various visual processing tasks. Specifically, through... This method acquires unlabeled global images or target entity sub-images of global images as sample images. Multiple trained second visual language models are then combined to generate diverse outputs based on prompt text. High-quality text is further filtered and text pairs are constructed. Finally, a dataset suitable for model training is formed based on the sample images and text pairs. This approach eliminates the need for manual image annotation, relying entirely on a large model to automatically generate supervision signals, thus automating the construction of training data, significantly reducing data preparation costs, and breaking the dependence on manual annotation. Furthermore, when acquiring target entity sub-images of global images, target detection is performed on the global image to obtain multiple detection boxes. The target entity sub-image is determined based on the semantic similarity of local image features of candidate entity sub-images within the detection boxes. This effectively removes duplicate and redundant information, avoiding wasted computational resources and processing result deviations caused by interference from similar entities. This allows the model to focus on key target entities, improving the accuracy of target recognition.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0011] Figure 1 This is a flowchart illustrating the training method of a multi-task visual model as shown in the first embodiment of this disclosure; Figure 2 This is a flowchart illustrating the training method of a multi-task visual model according to the second embodiment of this disclosure; Figure 3 This is a flowchart illustrating the training method for a multi-task visual model according to the third embodiment of this disclosure; Figure 4 This is a flowchart illustrating the training method for a multi-task visual model as shown in the fourth embodiment of this disclosure; Figure 5 This is a schematic diagram illustrating the principle of the training method for a multi-task visual model as shown in the embodiments of this disclosure; Figure 6 This is a schematic diagram of the structure of the visual language model shown in the embodiments of this disclosure; Figure 7 This is a schematic diagram of the structure of the multi-task vision model shown in the embodiments of this disclosure; Figure 8 This is a schematic diagram of the structure of a training device for a multi-task visual model as shown in the fifth embodiment of this disclosure; Figure 9 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present disclosure. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0013] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0014] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0015] With the development of artificial intelligence technology, deep learning-based visual model technology has made great progress and has been widely recognized in the industry. It has been implemented in multiple scenarios through the method of visual encoder (also known as base model) + model fine-tuning. The key to this problem is how to obtain a base model with high representation ability and low inference cost.

[0016] Early basic models were usually pre-trained for classification on large-scale image recognition datasets (such as ImageNet). Through fine-grained classification and diverse data composition, the models were able to achieve good representation results. However, as the application scenarios expanded, large-scale image recognition datasets gradually reached their bottlenecks, and the high cost of data annotation also prevented the basic models from improving further.

[0017] To eliminate the cost of data labeling, self-supervised learning-based pre-training methods for basic models have emerged. These methods optimize the representational ability of the basic model by using a large amount of unlabeled data through contrastive learning, mask restoration, self-distillation, and other techniques. Models trained in a self-supervised manner have better representational ability than classification pre-training and have achieved better performance in tasks such as few-shot, zero-shot, and similarity search compared to classification pre-training.

[0018] With the development of technology and the influence and impetus of Large Language Model (LLM) technology, Visual Language Model (VLM) has received more attention. On the one hand, the accumulation of multimodal datasets and the maturity of computing power have led to the gradual emergence of large-scale visual language models. These models have achieved near-human average performance in tasks such as text-to-image question answering, retrieval, and understanding, and have reached a new level in understanding visual semantics. Some models have also launched text-to-image question answering functions.

[0019] In related technologies, the following methods are mainly used to train the basic model: (1) Self-supervised method for mask restoration: Following the pre-training method of BERT, the image is divided into small patches of the same size, and a part is masked by uniform pixel values ​​in a random manner. The feature values ​​of the masked part are predicted by the unmasked image patches, and the base model is optimized by mean square error or cosine similarity loss. (2) Self-distillation basic model training: During the training process, a teacher model and a student model are constructed. The two models have completely identical structures. The parameters of the teacher model are updated by exponential moving average (EMA), while the student model is updated by backpropagation of error. During training, each input sample is transformed by cropping, rotation, normalization, etc. The samples before and after the transformation are input into the teacher model and the student model respectively. The loss function value is constructed by aligning the output feature values ​​of the two models. Based on the loss function value, the basic model is trained. (3) Weakly supervised algorithm: By constructing a dual-tower model, one side is a text encoder and the other side is a visual encoder. During training, the input is a text-image pair, that is, an image and a text description of the image content. The loss function value is constructed by aligning the text and image semantics. Based on the loss function value, the basic model is trained. (4) Training VLM: During training, a large amount of image-text question-and-answer pair data needs to be prepared, that is, one or more images and one or more image-text question-and-answer content. VLM is divided into three basic parts: visual encoder, large language model, and visual semantic adapter. During the training process, different data need to be used for multi-stage training, and different model parameters need to be systematically tuned to achieve performance optimization of visual encoder. While the aforementioned self-supervised masking methods have the lowest data annotation cost and can be trained as long as a sufficiently large dataset is available, the lack of semantic information in the text during training results in weak semantic representation capabilities, requiring more data for model training. For weakly supervised algorithms, a large amount of image-text pairs is needed. While this is abundant in internet scenarios, large-scale production of such data is impossible in financial, industrial, and manufacturing settings due to data security concerns, making such methods difficult to implement. As for Visual Models (VLMs), existing open-source multimodal large models offer excellent semantic question-answering performance, but their large size prevents direct integration into low-cost, high-concurrency visual analysis scenarios.

[0020] To address any of the above problems, this disclosure proposes a training method and apparatus for a multi-task visual model.

[0021] The following description, with reference to the accompanying drawings, describes a method and apparatus for training a multi-task visual model according to embodiments of the present disclosure.

[0022] Figure 1 This is a flowchart illustrating the training method for a multi-task visual model as shown in the first embodiment of this disclosure.

[0023] like Figure 1 As shown, the training method for this multi-task vision model includes: Step 110: Freeze the task decoder.

[0024] The multi-task vision model includes a first vision encoder and a task decoder.

[0025] In this embodiment of the disclosure, the multi-task vision model includes a first visual encoder and a task decoder. In order to avoid the interference of the feature learning process by the parameter changes of the task decoder, as a possible implementation, the task decoder in the multi-task vision model can be frozen, that is, the structure and parameters of the task decoder are kept unchanged, and the first visual encoder is trained and optimized independently.

[0026] Step 120: Obtain training data; wherein, the training data includes sample images and text pairs associated with the sample images.

[0027] To effectively train the first visual encoder and improve its feature extraction capability, one possible approach is to acquire training data containing sample images and text pairs associated with those sample images. The sample images refer to the raw image data used as input to the model, and may include visual content associated with the second target processing task (visual language processing task). The text pairs refer to a set of textual information semantically related to the sample images. Each text pair may include prompt text used to prompt the first visual language model to perform the first target processing task. The text pairs also include reference output text associated with the first target processing task. The second target processing task and the first target processing task may be the same visual language processing task or different visual language processing tasks; this disclosure does not impose specific limitations.

[0028] In some embodiments, such as Figure 2 As shown, step 120 may also include the following steps: Step 1201: Obtain sample images; wherein, the sample images include a global image associated with the second target processing task or a target entity sub-image in the global image, and the global image does not contain annotation information associated with the image task.

[0029] To construct training data suitable for a specific visual processing task, one possible approach is to acquire sample images. These sample images include a global image related to a second target processing task or a target entity sub-image cropped from the global image. The second target processing task can be a visual language processing task to be performed by the visual language model. The global image can be, for example, an entire scene image, and the target entity sub-image can be, for example, a local region of an object in the global image. Neither the global image nor the target entity sub-image contains manually labeled information related to the target task, such as category labels, bounding boxes, or segmentation masks.

[0030] In some embodiments, such as Figure 3 As shown, step 1201 may specifically include the following steps: Step 12011: Obtain the global image associated with the second target processing task.

[0031] As one possible implementation, raw images related to a second objective processing task are collected as global images, where the second objective processing task may include: image description generation, visual question answering, visual reasoning, etc. Step 12012: Perform target detection on the global image to obtain multiple detection boxes in the global image.

[0032] To achieve the decomposition from the global image to local regions, one possible approach is to automatically identify potentially interesting object regions in the global image using an object detection model and generate multiple detection boxes.

[0033] Step 12013: Determine the target entity sub-image from the candidate entity sub-images within each detection box based on the semantic similarity between the local image features of the candidate entity sub-images within each detection box in the global image.

[0034] To reduce interference from irrelevant regions, one possible approach is to calculate the semantic similarity between local image features of candidate entity sub-images within each detection box, and then select representative or most frequently occurring entity regions as target entity sub-images.

[0035] As an example, semantic features are extracted from candidate entity sub-images within each detection box in the global image to obtain local image features of candidate entity sub-images within each detection box; based on the semantic similarity between the local image features of candidate entity sub-images within each detection box, duplicate candidate entity sub-images within each detection box are deduplicated to obtain retained candidate entity sub-images; based on the retained candidate entity sub-images, the target entity sub-image is determined.

[0036] In other words, semantic feature extraction is performed on the candidate entity sub-images within each detection box in the global image to obtain the local image features corresponding to the candidate entity sub-images within each detection box. Then, based on the semantic similarity between these local image features, deduplication is performed on the candidate entity sub-images within each detection box. For example, if two or more candidate entity sub-images have high semantic similarity, it means that they are relatively close at the semantic level and there may be duplication. In this case, the duplicate sub-images will be removed, and finally, the retained candidate entity sub-images will be obtained. Then, these retained candidate entity sub-images are used as the target entity sub-images.

[0037] Step 12014: Select sample images from the global image and the target entity sub-image according to the preset sample distribution ratio.

[0038] It is important to understand that since the global image provides rich scene context information, it helps the model learn the semantic features of images in different scenes; while the target entity sub-image focuses on specific entities, it allows the model to better learn the detailed features of entities. Therefore, in order to improve the generalization ability of the model, a certain number of images are selected as samples from the global image and the target entity sub-image according to the set sample distribution ratio.

[0039] For example, if the sample distribution ratio is 60% global images and 40% target entity sub-images, when selecting samples, the corresponding number of images are extracted from the global image set and the target entity sub-image set respectively according to this ratio, and combined into the final sample image set.

[0040] In summary, by acquiring a global image associated with a visual processing task adapted to the multi-task visual model, object detection is performed on the global image to obtain multiple detection boxes. The target entity sub-image is determined based on the semantic similarity of local image features of candidate entity sub-images within the detection boxes. This effectively removes redundant and repetitive information, avoids wasted computational resources and processing result deviations caused by interference from similar entities, and allows the model to focus on key target entities, improving the accuracy of target recognition. Finally, sample images are selected from the global image and target entity sub-images according to a preset sample distribution ratio, ensuring the diversity and representativeness of the samples. This allows the model to learn the overall features of the scene and deeply grasp the detailed features of the target entities, thereby enhancing the model's adaptability and generalization performance to different types of images. This improves the processing effect and reliability of the multi-task visual model in various practical application scenarios.

[0041] Step 1202: Invoke multiple trained second visual language models to process the sample images based on the prompt text, so as to obtain the output text of multiple second visual language models.

[0042] It is important to understand that each trained second visual language model already possesses strong reasoning and language generation capabilities on large-scale image and text data. Therefore, in order to achieve multi-angle understanding of the semantic content of sample images and high-quality text generation, as a possible approach, multiple trained second visual language models can be used to generate multiple independent output texts for the same sample image and the same prompt text; among them, the prompt text is such as "Please describe this picture" or "What objects are in the picture?" etc.

[0043] Step 1203: Generate text pairs associated with the sample images based on the output text and prompt text of multiple second visual language models.

[0044] To construct high-quality training data, one possible approach is to evaluate the quality of the output text of each second visual language model, filter out high-quality output text, and combine it with prompt text to generate text pairs associated with sample images.

[0045] As an example, the output text of each second visual language model is quality evaluated to obtain a quality score for each output text; based on the output text and prompt text whose quality scores are higher than a set score threshold, text pairs associated with the sample image are generated.

[0046] To effectively improve the accuracy and reliability of automatically generated training data, one possible approach is to score the output text generated by each second visual language model using metrics such as semantic integrity, keyword coverage, logical consistency, or relevance to the prompt text, thus obtaining a corresponding quality score. Then, only high-quality texts with scores exceeding a preset threshold are retained, while ambiguous, erroneous, or redundant generated results are filtered out. Finally, these high-quality texts are used as reference outputs and paired with the corresponding prompt texts to generate text pairs.

[0047] To improve the accuracy of the quality assessment of the output text of each second visual language model, the following steps can be taken to assess the quality of the output text of each second visual language model: (1) For any output text of a second visual language, the quality of the output text of any second visual language is evaluated according to multiple set evaluation dimensions to obtain the quality sub-scores of the output text of any second visual language in each set evaluation dimension; In this embodiment of the disclosure, for the output text of any second visual language model, multiple predefined evaluation dimensions are determined. These dimensions refer to indicators that measure the quality of the output text from different perspectives. For example, the accuracy dimension indicates the degree to which the information conveyed by the output text matches the actual situation; the fluency dimension indicates whether the text sentences are fluent and whether there are obvious grammatical errors or awkward expressions; the richness dimension judges whether the text content is detailed and the amount of information is sufficient; and the relevance dimension determines the degree of correlation between the output text and the given visual input or task requirements. Based on these predefined evaluation dimensions, the output text of the second visual language model is analyzed, and quality sub-scores are given for each dimension according to the specific standards and requirements of each dimension. For example, in the accuracy dimension, if the output text is highly consistent with the real information, a higher score is given; if there are deviations or errors, points are deducted according to the degree of deviation.

[0048] (2) For each set evaluation dimension, the quality sub-scores of the output text of any second visual language model are weighted and summed to obtain the quality score of the output text of any second visual language model. Since different evaluation dimensions have different importance in measuring the overall quality of the output text, for example, in some scenarios where the accuracy of information is required to be high, the accuracy dimension may be more important than the fluency dimension. Therefore, corresponding weights are set for each evaluation dimension. These weights reflect the relative importance of each dimension in the overall quality evaluation. Then, by multiplying the quality sub-score of each dimension by the corresponding weight and summing all the weighted sub-scores, the quality score of the output text of the second visual language model can be obtained.

[0049] For example, if the accuracy dimension has a weight of 0.5, the corresponding quality sub-score is 8; the fluency dimension has a weight of 0.3, the quality sub-score is 7; and the richness dimension has a weight of 0.2, the quality sub-score is 6. Then the quality score = 8×0.5 + 7×0.3 + 6×0.2 = 4 + 2.1 + 1.2 = 7.3.

[0050] Step 1204: Generate training data based on the sample images and text pairs.

[0051] Then, the text pairs are paired with the corresponding sample images to generate training data.

[0052] In summary, by acquiring unlabeled global images or target entity sub-images of global images as sample images, and combining multiple trained second visual language models to generate diverse outputs based on prompt text, high-quality texts are further filtered and text pairs are constructed. Finally, a dataset that can be used for model training is formed based on sample images and text pairs. This achieves the goal of automatically generating supervision signals entirely by relying on large models without the need for manual annotation of image content, thus realizing the automated construction of training data, significantly reducing data preparation costs, and breaking through the dependence on manual annotation.

[0053] Step 130: Using the language decoder in the trained first visual language model, train the first visual encoder based on sample images and text pairs in the training data.

[0054] To achieve effective training of the first visual encoder, one possible approach is to utilize the language decoder in the trained first visual language model to construct a cross-modal supervision signal based on sample image and text pairs in the training data, and then train the first visual encoder.

[0055] The first visual language model also includes a second visual encoder, which has fewer parameters than the second visual encoder; the task decoder has fewer parameters than the language decoder. As a result, the multi-task visual model is more lightweight, has faster inference speed, and consumes less computing resources.

[0056] In some embodiments, such as Figure 4 As shown, step 130 may include the following steps: Step 1301: Input the sample image into the first visual encoder to obtain the image semantic features of the sample image.

[0057] To extract semantically meaningful visual representations from sample images, one possible approach is to input the sample images into a first visual encoder to obtain the image semantic features of the sample images. The first visual encoder performs block segmentation, embedding, and self-attention modeling on the sample images, converting the original pixel data of the sample images into semantic features in the form of high-dimensional vectors, thus obtaining the image semantic features of the sample images output by the first visual encoder.

[0058] Step 1302: The language decoder of the trained first visual language model performs target processing tasks on the semantic features of the image based on the prompt text to obtain the predicted output text.

[0059] To achieve semantic-driven processing of image semantic features, one possible approach is to use a language decoder of a trained first visual language model. This language decoder leverages the language knowledge and task understanding learned from large-scale image and text data to decode image semantic features guided by prompt text, generating natural language output that conforms to semantic logic, i.e., the predicted output text.

[0060] Step 1303: Determine the loss function value based on the difference between the predicted output text and the reference output text.

[0061] To accurately optimize model parameters, one possible approach is to calculate the semantic or lexical inconsistency between the predicted output text and the reference output text, and quantify this inconsistency into a scalar loss value. For example, cross-entropy loss, loss combined with BLEU scores, or semantic similarity loss can be used. It should be noted that this loss function value reflects the degree of deviation between the model's current output and the expected output, and serves as a training signal for backpropagation to guide model parameter updates.

[0062] Step 1304: Train the first visual encoder based on the loss function value.

[0063] To optimize image feature extraction capabilities, one possible approach is to use the backpropagation algorithm to pass the gradient corresponding to the loss function value to the first visual encoder, update the network parameters of the first visual encoder, and enable the first visual encoder to generate image semantic features that better meet the task requirements in subsequent forward computation.

[0064] As an example, semantic features are extracted from the predicted output text to obtain the first semantic features of the predicted output text; semantic features are extracted from the reference output text to obtain the second semantic features of the reference output text; and the loss function value is determined based on the difference between the similarity between the first semantic features and the second semantic features and the target similarity.

[0065] In other words, in order to train a more semantically accurate first visual encoder, semantic features are extracted from the predicted output text to obtain its first semantic features. At the same time, semantic features are extracted from the reference output text to obtain the second semantic features. The semantic similarity (such as cosine similarity) between the first and second semantic features is calculated, and the calculated similarity is compared with the target similarity to obtain the difference between the similarity and the target similarity. Based on the difference, a loss function value is constructed, and then the first visual encoder is trained based on the loss function value.

[0066] The loss function value is positively correlated with the above-mentioned differences; that is, the smaller the difference, the smaller the loss function value, and vice versa. Thus, in this disclosure, the first visual encoder can be trained based on the loss function value to minimize the loss function value.

[0067] It should be noted that the above example only uses minimizing the loss function value as the termination condition for training the first visual encoder. In actual applications, other termination conditions can also be set, such as the number of training iterations reaching a set threshold, the training duration exceeding a set duration threshold, etc. This disclosure does not impose any restrictions on these conditions.

[0068] In summary, by inputting sample images into the first visual encoder to extract semantic features, and using the language decoder of the trained first visual language model in combination with the prompt text to perform task-driven decoding of the image features, predictive output text is generated. Then, the loss function value is calculated based on the difference between the predicted text and the reference text, thereby optimizing the first visual encoder in reverse. Thus, the rich cross-modal semantic knowledge of the large model is fully utilized as a supervision signal, and the first visual encoder is effectively trained without the need for manual annotation of complex labels.

[0069] The training method for the multi-task visual model in this disclosure avoids the complexity and uncertainty caused by adjusting the parameters of the task decoder during training by pre-freezing the task decoder during the training process. This allows the training focus to be highly concentrated on the first visual encoder. Then, by using the language decoder in the trained first visual language model, combined with training data containing sample images and their associated text pairs, the first visual encoder is trained. This fully utilizes the rich language features and semantic understanding capabilities already learned by the language decoder, enabling the first visual encoder to learn the association information between images and text. This targeted training method helps improve the first visual encoder's ability to extract image features, allowing it to more accurately capture key information in images. This provides higher-quality feature representations for subsequent task decoders, significantly improving the accuracy and robustness of the multi-task visual model in various visual processing tasks.

[0070] Based on any of the above embodiments, such as Figure 5 As shown, the training method for the multi-task vision model in this embodiment may include the following steps: Step 510, Vertical Industry Unlabeled Dataset First, prepare an unlabeled dataset for the vertical industry. After a pre-labeling process for training data, a training dataset for model training is formed. The specific steps are as follows: Step 5101, Image data preprocessing By introducing a detection model trained on a dataset containing multiple objects (such as yolov5-obj365), the locations of entity sub-images contained in the global image are obtained. Based on the detection box information, sub-images are cropped. Data deduplication is performed by comparing contrastive language-image pre-training (CLIP) features. Based on the sample distribution, the ratio of the global image to the sub-image is controlled at an appropriate ratio (such as 1:5). In this way, the resulting visual model is guaranteed to have good representation ability for both the entire set of images and local images.

[0071] Step 5102, Production of training set based on open source VLM Take the prepared dataset, call the open-source VLM model in sequence, and randomly select prompt text, such as "Describe the entities in the image and the relationships between the entities", "Explain the main entities in the image and the relationships between the secondary entities", "Describe the semantic content in the image, and describe it as finely as possible", etc.

[0072] 5103, Dataset Sampling and Filtering For the generated dataset, a sampling quality check is performed. Since the open-source VLM models are trained on different datasets, some models may have low accuracy for the description content of the current vertical industry. By using the sampling accuracy statistics, the results produced by some models can be excluded from the training set.

[0073] The training set for model training is obtained through the above steps. The training set consists of images and their question-and-answer pairs. Each image carries one or more text question-and-answer pairs. For each image from image set I... ,Right now There is a set of question-answer pairs Together, they constitute the training dataset: .

[0074] Step 520, Model Training like Figure 6 As shown, the VLM model is decomposed into two parts: a visual encoder with a Vision Transformer (ViT) structure. Input image Obtaining semantic features of images The relevant parameters are A large language model (LLM) Input image semantic features and related issues Output description The relevant parameters are .

[0075] During model training, first select an open-source VLM model and obtain its LLM model parameters. To address the computational limitations of the deployed model, a new visual encoder will be created. The parameters are ,because Strong versatility is not required. The number of parameters is usually greater than The value is about an order of magnitude smaller, and the loss function value is determined using cross-entropy, with the specific formula as follows: Loss ; The model training objective is to optimize The parameters minimize the loss function value, i.e. The visual encoder obtained in this way It has strong visual representation capabilities in the video input domain of vertical industries.

[0076] Furthermore, such as Figure 7 As shown, downstream application algorithm models can be quickly developed by connecting different algorithm task decoders downstream. Since the ViT algorithm framework is used, the basic model can be adjusted by lightweight model fine-tuning techniques (such as Adapter and LoRa).

[0077] Corresponding to the multi-task visual model training method provided in the above embodiments, this disclosure also provides a multi-task visual model training device. Since the multi-task visual model training device provided in this disclosure corresponds to the multi-task visual model training method provided in the above embodiments, the implementation of the business processing method is also applicable to the multi-task visual model training device provided in this disclosure, and will not be described in detail in this disclosure.

[0078] Figure 8 This is a schematic diagram of the structure of a training device for a multi-task visual model according to the fifth embodiment of this disclosure. The multi-task visual model includes a first visual encoder and a task decoder.

[0079] like Figure 8 As shown, the training device 800 for the multi-task visual model includes: a freezing module 810, an acquisition module 820, and a training module 830.

[0080] The freeze module 810 is used to freeze the task decoder; the acquisition module 820 is used to acquire training data; wherein the training data includes sample images and text pairs associated with the sample images; and the training module 830 is used to train the first visual encoder based on the sample images and text pairs in the training data using the language decoder in the trained first visual language model.

[0081] As one possible implementation, the text pair includes a cue text used to prompt the first visual language model to perform a first target processing task. The text pair also includes a reference output text associated with the first target processing task. The training module 830 is used to input the sample image into the first visual encoder to obtain the image semantic features of the sample image; the language decoder of the trained first visual language model performs the first target processing task on the image semantic features based on the cue text to obtain the predicted output text; the loss function value is determined according to the difference between the predicted output text and the reference output text; and the first visual encoder is trained according to the loss function value.

[0082] As one possible implementation, the training module 830 is used to extract semantic features from the predicted output text to obtain the first semantic features of the predicted output text; extract semantic features from the reference output text to obtain the second semantic features of the reference output text; and determine the loss function value based on the difference between the similarity between the first semantic features and the second semantic features and the target similarity.

[0083] As one possible implementation, the acquisition module 820 is used to acquire sample images; wherein, the sample images include a global image associated with the second target processing task or a target entity sub-image in the global image, and the global image does not contain annotation information associated with the image task; multiple trained second visual language models are invoked to perform the second target processing task on the sample images based on prompt text to obtain the output text of multiple second visual language models; text pairs associated with the sample images are generated based on the output text of multiple second visual language models and prompt text; training data is generated based on the sample images and text pairs.

[0084] As one possible implementation, the acquisition module 820 is used to perform quality assessment on the output text of each second visual language model to obtain a quality score for each output text; and to generate text pairs associated with the sample image based on the output text and prompt text whose quality scores are higher than a set score threshold.

[0085] As one possible implementation, the acquisition module 820 is used to acquire a global image associated with the second target processing task adapted to the multi-task visual model; perform target detection on the global image to obtain multiple detection boxes in the global image; determine the target entity sub-image from the candidate entity sub-images in each detection box based on the semantic similarity between the local image features of the candidate entity sub-images in each detection box in the global image; and select sample images from the global image and the target entity sub-image according to a preset sample distribution ratio.

[0086] As one possible implementation, the acquisition module 820 is used to extract semantic features from the candidate entity sub-images within each detection box in the global image to obtain the local image features of the candidate entity sub-images within each detection box; based on the semantic similarity between the local image features of the candidate entity sub-images within each detection box, the candidate entity sub-images within each detection box are deduplicated to obtain the retained candidate entity sub-images; and based on the retained candidate entity sub-images, the target entity sub-image is determined.

[0087] As one possible implementation, the acquisition module 820 is used to perform quality assessment on the output text of any second visual language model according to multiple set evaluation dimensions to obtain the quality sub-scores of the output text of any second visual language model in each set evaluation dimension; and to perform weighted summation of the quality sub-scores of the output text of any second visual language model according to the weights of each set evaluation dimension to obtain the quality score of the output text of any second visual language model.

[0088] As one possible implementation, the first visual language model also includes a second visual encoder, where the number of parameters of the first visual encoder is less than the number of parameters of the second visual encoder; the number of parameters of the task decoder is less than the number of parameters of the language decoder. The training apparatus for the multi-task visual model in this disclosure avoids the complexity and uncertainty caused by adjusting the parameters of the task decoder during training by pre-freezing the task decoder during the training process. This allows the training focus to be highly concentrated on the first visual encoder. Then, by using the language decoder in the trained first visual language model, combined with training data containing sample images and their associated text pairs, the first visual encoder is trained. This fully leverages the rich linguistic features and semantic understanding capabilities already learned by the language decoder, enabling the first visual encoder to learn more accurate and in-depth association information between images and text. This targeted training method helps improve the first visual encoder's ability to extract image features, allowing it to more accurately capture key information in images. This provides higher-quality feature representations for subsequent task decoders, significantly improving the accuracy and robustness of the multi-task visual model in various visual processing tasks.

[0089] In an exemplary embodiment, an electronic device is also proposed.

[0090] The electronic devices include: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the training method for the multi-task vision model as proposed in any of the foregoing embodiments.

[0091] As an example, Figure 9 This is a schematic diagram of the structure of an electronic device 900 as shown in an exemplary embodiment of this disclosure, as follows: Figure 9 As shown, the aforementioned electronic device 900 may further include: The present invention includes a memory 910 and a processor 920, and a bus 930 connecting different components (including the memory 910 and the processor 920). The memory 910 stores a computer program, and when the processor 920 executes the program, it implements the training method of the multi-task visual model described in the embodiments of the present disclosure.

[0092] Bus 930 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0093] Electronic device 900 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 900, including volatile and non-volatile media, removable and non-removable media.

[0094] The memory 910 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 940 and / or cache memory 950. The electronic device 900 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 960 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 9 Not shown; usually referred to as a "hard drive"). Although Figure 9As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 1130 via one or more data media interfaces. Memory 910 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0095] A program / utility 980 having a set (at least one) of program modules 970 may be stored, for example, in memory 910. Such program modules 970 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 970 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0096] Electronic device 900 can also communicate with one or more external devices 990 (e.g., keyboard, pointing device, display 991, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 992. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 993. As shown, network adapter 993 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0097] The processor 920 executes various functional applications and data processing by running programs stored in the memory 910.

[0098] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the training method of the multi-task visual model in the present disclosure, and will not be repeated here.

[0099] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device to complete the training method of the multi-task visual model proposed in any of the above embodiments. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0100] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the training method for the multi-task visual model proposed in any of the above embodiments.

[0101] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0102] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a multi-task visual model, characterized in that, The multi-task vision model includes a first visual encoder and a task decoder, including: Freeze the task decoder; Acquire training data; wherein the training data includes sample images and text pairs associated with the sample images; the acquisition of training data includes: acquiring sample images; wherein the sample images include a global image associated with a second target processing task or a target entity sub-image in the global image, wherein the global image does not contain annotation information associated with the second target processing task; calling multiple trained second visual language models, performing a second target processing task on the sample images based on prompt text, to obtain output text of multiple second visual language models; generating text pairs associated with the sample images based on the output text of the multiple second visual language models and the prompt text; generating the training data based on the sample images and the text pairs; The first visual encoder is trained using the language decoder in the trained first visual language model, based on the sample images and text pairs in the training data.

2. The method according to claim 1, characterized in that, The text pair includes prompt text, wherein The prompt text is used to prompt the first visual language model to perform a first target processing task, and the text pair also includes: reference output text associated with the first target processing task; The step of training the first visual encoder using the language decoder in the trained first visual language model, based on the sample images and text pairs in the training data, includes: The sample image is input into the first visual encoder to obtain the image semantic features of the sample image; The language decoder of the first visual language model trained thereon performs the first target processing task on the semantic features of the image based on the prompt text, and obtains the predicted output text; The loss function value is determined based on the difference between the predicted output text and the reference output text; The first visual encoder is trained based on the loss function value.

3. The method according to claim 2, characterized in that, The step of determining the loss function value based on the difference between the predicted output text and the reference output text includes: Semantic features are extracted from the predicted output text to obtain the first semantic features of the predicted output text; Semantic features are extracted from the reference output text to obtain the second semantic features of the reference output text; The loss function value is determined based on the difference between the similarity between the first semantic feature and the second semantic feature and the target similarity.

4. The method according to claim 1, characterized in that, The step of generating text pairs associated with the sample image based on the output text of the plurality of second visual language models and the prompt text includes: The output text of each of the second visual language models is evaluated for quality to obtain a quality score for each output text. Based on the output text whose quality score is higher than a set score threshold and the prompt text, a text pair associated with the sample image is generated.

5. The method according to claim 1, characterized in that, The acquisition of sample images includes: Obtain the global image associated with the second target processing task; Target detection is performed on the global image to obtain multiple detection boxes in the global image; Based on the semantic similarity between the local image features of the candidate entity sub-images within each detection box in the global image, the target entity sub-image is determined from the candidate entity sub-images within each detection box; The sample image is selected from the global image and the target entity sub-image according to the preset sample distribution ratio.

6. The method according to claim 5, characterized in that, The step of determining the target entity sub-image from the candidate entity sub-images within each detection box based on the semantic similarity between the local image features of each candidate entity sub-image within each detection box in the global image includes: Semantic features are extracted from the candidate entity sub-images within each detection box in the global image to obtain the local image features of the candidate entity sub-images within each detection box. Based on the semantic similarity between the local image features of the candidate entity sub-images within each detection box, the candidate entity sub-images within each detection box are deduplicated to obtain the retained candidate entity sub-images. The target entity subgraph is determined based on the retained candidate entity subgraphs.

7. The method according to claim 4, characterized in that, The quality evaluation of the output text of each of the second visual language models to obtain a quality score for each output text includes: For the output text of any second visual language model, the quality of the output text of the second visual language model is evaluated according to multiple set evaluation dimensions to obtain the quality sub-scores of the output text of the second visual language model in each set evaluation dimension; The quality sub-scores of the output text of any second visual language model are weighted and summed according to the weights of each of the defined evaluation dimensions to obtain the quality score of the output text of any second visual language model.

8. The method according to claim 1, characterized in that, The first visual language model further includes a second visual encoder, the number of parameters of the first visual encoder being less than the number of parameters of the second visual encoder; the number of parameters of the task decoder is less than the number of parameters of the language decoder.

9. A training device for a multi-task visual model, characterized in that, The multi-task vision model includes a first visual encoder and a task decoder, including: A freeze module is used to freeze the task decoder; An acquisition module is used to acquire training data; wherein the training data includes sample images and text pairs associated with the sample images; the acquisition of training data includes: acquiring sample images; wherein the sample images include a global image associated with a second target processing task or a target entity sub-image in the global image, wherein the global image does not contain annotation information associated with the second target processing task; invoking multiple trained second visual language models to perform a second target processing task on the sample images based on prompt text, so as to obtain the output text of multiple second visual language models; generating text pairs associated with the sample images based on the output text of the multiple second visual language models and the prompt text; and generating the training data based on the sample images and the text pairs. The training module is used to train the first visual encoder based on the sample images and text pairs in the training data, using the language decoder in the trained first visual language model.

Images