Multi-task learning method for visual inspection based on multimodal cue collaboration

Abstract

The present invention discloses a visual inspection multi-task learning method based on multimodal prompt collaboration, and uses a multimodal prompt collaborative multi-task learning network as a visual-language model to implement visual inspection multi-task learning. The multimodal prompt collaborative multi-task learning network includes a language prompt encoder and a visual encoder, and the language prompt encoder includes a parallel frozen language encoder and a language encoder; the frozen language encoder is used to ensure that the adjustment of the language encoder is not greater than a threshold limit; the language encoder and the visual encoder use a coupling function to calculate the cosine similarity of the features of the two to measure the similarity of the two tasks, so as to maximize the consistency of the visual and language representations of the task group with high total similarity, and realize efficient joint training of visual inspection multi-task learning. The present invention improves the adjustment effect of the visual inspection multi-task model and ensures the pertinence and generalization of the visual inspection multi-task learning method.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual inspection, and in particular relates to a multi-task learning method for visual inspection based on multi-modal prompt collaboration. Background Art

[0002] Visual inspection has always been an important means of guarantee for industrial production, operation and maintenance management, and detection and monitoring. Traditional visual inspection has always relied on manual inspection methods, which are greatly affected by time and space, with high labor intensity, low efficiency and high cost. Especially in some environments with complex natural conditions or production conditions, manual inspection methods are difficult to implement, or even if manual inspection is applied, it is difficult to obtain ideal inspection results. With the rapid development of robotics technology, the use of inspection robots to replace manual inspection tasks in long-term and complex environments has become the main method at present. The increasingly complex production environment requires inspection robots to continuously improve their intelligence and multi-tasking levels. In recent years, the rapid development of deep learning technology has led to a significant leap in various technologies in the field of artificial intelligence, especially in the fields of natural language processing and computer vision. A large number of high-efficiency and high-precision natural language processing, target detection and recognition, scene segmentation, scene understanding and other methods have emerged. Visual inspection robots are also gradually using artificial intelligence algorithms to replace traditional machine learning methods to adapt to more complex application scenarios.

[0003] At present, visual inspection robots use computer vision algorithms to simulate the working mode of the human visual system to perceive, understand and process the external environment. However, the data that a single visible light visual data can provide is limited, and the visual inspection method using a single visual source (such as visible light images, depth images or infrared images) is usually susceptible to environmental interference, and the visual inspection method using multiple visual sources will encounter inevitable detection confidence problems during the inspection process. At present, large models have performed well in industrial applications. The use of multimodal prompt collaborative methods for inspection task learning is a suitable method for integrated detection of multiple data sources. At present, multimodal prompt collaborative learning generally adopts a separate feature extractor and a multimodal feature fusion module. Prompting a single branch for adjustment cannot achieve the optimal effect, and the detection performance of unknown categories is reduced. Summary of the invention

[0004] The technical problem to be solved by the present invention is how to overcome the shortcomings of large differences in multimodal feature distribution and insufficient optimization effect of a single branch, so as to improve the model adjustment effect for multiple tasks of visual inspection and ensure the detection performance of unknown categories in complex scenarios.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A visual inspection multi-task learning method based on multimodal cue collaboration, using a multimodal cue collaborative multi-task learning network as a visual-language model to implement visual inspection multi-task learning, wherein the multimodal cue collaborative multi-task learning network includes a language cue encoder and a visual encoder, and the language cue encoder includes a parallel frozen language encoder and a language encoder;

[0007] Freezing the speech encoder to ensure that the adjustment of the speech encoder is not greater than a threshold limit;

[0008] The language encoder and the visual encoder measure the similarity of the two tasks by calculating the cosine similarity of their features through a coupling function, so as to maximize the consistency of the visual and language representations of the task group with high total similarity and realize efficient joint training of multi-task learning of visual inspection.

[0009] To optimize the above technical solutions, the specific measures taken also include:

[0010] The above method comprises the following steps:

[0011] Step 1: Set the parameters of the multi-modal prompt collaborative multi-task learning network;

[0012] Step 2: Freeze the pre-trained visual-language model, given the initial shared source cues for all tasks, calculate the gradient cosine similarity of the language encoder and the visual encoder for the shared cue parameters of the two tasks through the coupling function, measure the similarity of the two tasks, and group the tasks in the visual inspection multi-task training dataset;

[0013] Step 3: For each group of tasks, provide group-initial source prompts and task-initial source prompts, combine with the scaling matrix, and generate prompts for the language encoder and visual encoder;

[0014] Step 4: Based on the cues from the language encoder and the visual encoder, the training set of each task is divided into multiple batches for training, where each training batch contains several language-visual cue pairs.

[0015] The above step 2 calculates the gradient cosine similarity of the shared prompt parameters of the language encoder and the visual encoder through the coupling function, as follows:

[0016]

[0017] in, represents the gradient cosine similarity, Represents two different tasks for language encoder and visual encoder, Indicates the initial shared source prompt; represents the gradient of the initial shared source cue; represents the cross entropy loss function.

[0018] The prompts of the language encoder and the visual encoder in step 3 above are generated as follows:

[0019]

[0020]

[0021]

[0022]

[0023] in, , The language and visual cues for a single task are for the language encoder and the visual encoder, respectively; , are the initial language source cue and visual source cue for the language encoder and visual encoder, respectively; represents the Kronecker product operation; and They are group-initial source cues and task-initial source cues, respectively; They correspond respectively to the language encoder group prompt scaling matrix, the visual encoder group prompt scaling matrix, the language encoder group task prompt scaling matrix, and the visual encoder group task prompt scaling matrix.

[0024] In step 4 above, for language prompts, the language prompts are first projected into the word vector space after lexical distraction to obtain , represents the length of the word vector;

[0025] Encode each language prompt into length, and then through a vector of length The language encoder is transformed, and finally we get The speech encoder output ;

[0026] At each transformation layer of the speech encoder, The vector conversion is as follows:

[0027]

[0028]

[0029] in, Indicates the concatenation operation of parameters;

[0030] and Represents two consecutive layers of tokens in the language encoder model. Indicates the connected language encoder The transformation function of the layer, Indicates the language encoder Layer Group Tips, Indicates the language encoder Layer task prompts, and Corresponding to the language encoder Layer group prompt scaling matrix, task prompt scaling matrix, and represent group source cues and task source cues, respectively;

[0031] Finally, through the projection function Will Projecting to the latent embedding space of the visual-language model to obtain the final language representation .

[0032] In step 4 above, for the visual input, the block vectorization of the visual input is first performed, in which the aligned visual image, infrared image and depth image are used as the joint visual input image. The three visual inputs are connected in the channel dimension, and the input image format is ,in represents the matrix height and length, Indicates the number of channels of the image; the input image is reorganized into a set of input 2D image blocks, the image blocks , the size of the image block is , the number of image blocks , and then use a trainable linear projection function to transform the image patch to dimensional potential vector, complete block vectorization, calculated as follows:

[0033]

[0034] in, Represents different image blocks, represents the image vectorization operation, Represents image position vectorization operations Represents the dimensional space where the data is located.

[0035] Then with the learnable category labels Combined into , input into the visual encoder.

[0036] Each layer of the above-mentioned visual encoder is composed of a multi-head self-attention module MSA and a multi-layer perceptron MLP. Layer normalization is performed at the beginning of each layer. A residual connection is used between MSA and MLP. The multi-layer perceptron MLP contains two GELU nonlinear activation layers, which are superimposed in sequence to visual encoder layer; for the layer, is passed to Layers are processed sequentially by the following transformers:

[0037]

[0038]

[0039]

[0040]

[0041] in, Indicates Class Token and image words The connection vector of yes and The intermediate variable of calculation, Represents the visual encoder The layer's group prompt, Represents the visual encoder Individual task prompts for each layer. They represent the visual encoder The layer's group prompt scaling matrix, task prompt scaling matrix, Respectively represent The group source hint and task source hint of the layer, represents the normalized result, and Indicates The category words and image words of the layer;

[0042] , They are self-attention, normalization, and multi-layer perception functions respectively.

[0043] Individual task visual cues for the above visual encoders During the training process, the language encoder and the visual encoder are dynamically updated according to the prompts of the previous layer. The calculation method can be expressed as follows:

[0044]

[0045] in, is the balance weight, control dynamic update, normalized exponential function Control the output probability and be 1, Represents the visual encoder Layer individual task prompts, Represents the first Layer individual task prompts.

[0046] After the encoding conversion of the visual encoder, the image obtained by the last encoder layer is marked as , using the image projection function Project it into the latent embedding space of the visual-language model to obtain the final visual representation: .

[0047] The loss function used in the training process of step 4 above is:

[0048]

[0049]

[0050]

[0051]

[0052] in, They are group shared cue parameter loss function, single task cue parameter loss function, and language encoder parameter adjustment loss function; represents the concatenation of the group cue and the task cue in the language encoder, represents the concatenation of the group-initial source prompt and the task-initial source prompt, and They are group-initial source cues and task-initial source cues, respectively; They correspond to the language encoder group prompt scaling matrix, the visual encoder group prompt scaling matrix, the language encoder task prompt scaling matrix, and the visual encoder task prompt scaling matrix, respectively. is the penalty weight, which controls the magnitude of the weight adjustment after the language encoder is adjusted during training. Indicates the number of language encoder layers.

[0053] The present invention has the following beneficial effects:

[0054] The present invention designs a visual inspection multi-task learning method that includes multi-modal input of language, visible light image, infrared image and depth image, expands the multi-modal input type boundary of visual inspection, and simultaneously performs multi-task network model learning for different data sources, which can fuse multi-sensor data to solve different problems at the same time: for multi-task learning in visual inspection, a multi-modal prompt collaborative learning framework is adopted to address the shortcomings of single visual source and multi-visual source methods such as susceptibility to interference and difficulty in fusion and trust, and multi-modal data is used to make up for the single defect of single visible light visual data, and reasonable language prompts are provided for visual data. Combined with the advantages of large language models in large-scale knowledge accumulation, transferable learning and parameter fine-tuning, a multi-modal vision-language model is constructed with multi-modal visual data such as visible light images, depth images, infrared images, etc., to realize a visual inspection multi-task learning method, improve the application scenario range and detection accuracy of inspection robots, and enhance the reliability and accuracy of visual inspection. The present invention adopts a pre-trained large language / vision model and a multimodal prompt collaboration method to improve the consistency of visible light images, depth images and infrared images with language representation, promote the correlation between visual prompts and language prompts, ensure the mutual collaboration between the two, and ultimately achieve multi-task learning of visual inspection tasks.

[0055] In view of the particularity of visual inspection tasks, the present invention proposes a homogeneous fusion multimodal prompt collaborative learning framework to adjust both language prompts and visual prompts at the same time, enhance the correlation between the two, and constrain the adjustment degree of the language encoder to ensure that the model does not only optimize the language prompt content, but also the detection performance decreases for the non-appearing content. Ultimately, the adjustment effect of the visual inspection multi-task model is improved to ensure the pertinence and generalization of the visual inspection multi-task learning method.

[0056] The present invention designs a frozen adjustment loss function between the pre-trained visual-language model and the learnable language, constrains the adjustment range of the language model prompt, avoids overfitting of the model to certain language prompts during the training process, resulting in reduced generalization of the visual-language model, and ensures the generalizability of the multi-task learning results.

[0057] The present invention designs a learnable shared source prompt and a learnable scaling matrix to enhance the consistency of the learnable prompts of the language encoder and the visual encoder; and generates group shared prompts and individual task prompts based on the shared source prompts to achieve consistent improvement in single-task learning and intra-group task learning performance. The multimodal prompt collaborative training of the entire network training process relies on the above two operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a diagram of the network structure of visual inspection multi-task learning based on multi-modal prompt collaboration of the present invention;

[0059] Figure 2This is a network structure diagram of the visual encoder layer of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0061] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0062] See also Figure 1 The visual inspection multi-task learning method based on multimodal cue collaboration of the present invention uses a multimodal cue collaborative multi-task learning network as a visual-language model to realize visual inspection multi-task learning. The designed visual inspection multi-task learning network based on multimodal cue collaboration can be divided into two parts according to function: a language cue encoder and a visual cue encoder, and the two are associated with consistent cue parameters using a coupling function. In the language-visual cue collaborative learning process, a multimodal method is used to adapt to the visual-language dual branches: the language cue encoder is divided into two parallel encoder modules: a frozen language encoder and a language encoder, and the difference in weights after the frozen language encoder and the language encoder are constrained to ensure that the adjustment of the language encoder is not greater than the threshold limit, so as to prevent overfitting for specific tasks; the language encoder and the visual cue encoder measure the similarity of the two tasks by calculating the cosine similarity of their features, that is, the coupling function is used to ensure that the language encoder and the visual encoder have the same weight. , , , The consistency of , The language and visual cues for a single task are respectively for the language encoder and the visual encoder, , They are the initial language source cues and visual source cues for the language encoder and visual encoder, respectively. The coupling function includes global parameter consistency and grouped task similarity calculation. First, the frozen pre-trained visual-language model is used to calculate the cosine similarity between tasks, and tasks are grouped according to the similarity. For each group of tasks, the initial shared source cue parameters and the language cues and visual cues for a single task are given. Finally, the group shared cue parameter loss function, the single task cue parameter loss function and the language encoder parameter adjustment loss function are jointly optimized to achieve the purpose of multi-task model training. It can realize simultaneous training of multiple tasks in visual inspection, and determine the consistency of visual and language representations of task groups with high total similarity to maximize the consistency, so as to realize efficient joint training of multi-task learning in visual inspection.

[0063] Specifically, the visual inspection multi-task learning method based on multimodal prompt collaboration of the present invention includes:

[0064] Step 1: Set the parameters of the multi-modal prompt collaborative multi-task learning network;

[0065] In the embodiment, it is necessary to prepare a visual inspection multi-task dataset, a pre-trained visual-language model, and an initial sharing prompt ;

[0066] The basic settings of the network are as follows: CLIP (ViT-B / 16) is selected as the default model, a batch size of 32 / 4 is used, 5 epochs are trained, a stochastic gradient descent optimizer is used, and the learning rate is set to 0.005. The size of the language encoder , the size of the image encoder is .

[0067] Step 2: Freeze the pre-trained visual-language model, given the initial shared source cues for all tasks, calculate the gradient cosine similarity of the language encoder and the visual encoder for the shared cue parameters of the two tasks through the coupling function, measure the similarity of the two tasks, and group the tasks in the visual inspection multi-task training dataset;

[0068] In the embodiment, the pre-trained visual-language model is frozen, and given the source prompt parameters that are globally shared by all tasks, the similarity between the two tasks is measured by calculating the gradient cosine similarity of the shared prompt parameters of the two tasks, and then the grouping result is obtained. The calculation method is as follows:

[0069]

[0070] in, represents the gradient cosine similarity, Represents two different tasks. Indicates the initial shared source hint, represents the gradient of the initial shared source cue, Loss function.

[0071] In the actual grouping process, we first freeze the pre-trained visual-language model and use the initial shared source prompts. Calculate the gradient cosine similarity of all tasks on the shared parameters, the initial shared source hint It is only used for task grouping and not for subsequent training. The gradients of different layers are used for comprehensive similarity calculation. Finally, all the original multi-task datasets are grouped according to the similarity. The tasks are divided into different task groups.

[0072] Step 3: For each group of tasks, provide group initial source prompts and task initial source prompts for each task, and combine with the scaling matrix to generate prompts for the language encoder and the visual encoder;

[0073] In the embodiment, for each group of tasks, a group initial source prompt is provided and the task initial source prompt , and then use the Kronecker product to generate prompts for the language encoder and the visual encoder , , , ,in, , The language and visual cues for a single task are for the language encoder and the visual encoder, respectively; , are the initial language source cues and visual source cues for the language encoder and visual encoder, respectively.

[0074] During initialization, and Consistent, and Consistent.

[0075] Group initial source prompt , language encoder tips , Video Encoder Tips , then two scale matrices are defined to convert the initial source prompts to language prompts and visual prompts. The scale matrix is ​​defined as , which can be described in mathematical form as follows:

[0076]

[0077]

[0078]

[0079]

[0080] in, represents the Kronecker product operation, and are the group-initial source prompt and the task-initial source prompt, They correspond respectively to the language encoder group prompt scaling matrix, the visual encoder group prompt scaling matrix, the language encoder group task prompt scaling matrix, and the visual encoder group task prompt scaling matrix.

[0081] Step 4: Based on the language encoder cues and the visual encoder cues, the training set of each task is divided into multiple batches for training, where each training batch contains several language-visual cue pairs.

[0082] In the embodiment, the training set of each task group is divided into multiple batches for training, and each training batch contains M (language-vision) pairs.

[0083] First, the language prompts are processed and projected into the word embedding space after lexical distraction to obtain , which represents the length of the word vector. Each language prompt is encoded into length, and then through a vector of length The language encoder prompts the conversion, and finally gets The speech encoder output At each conversion layer of the speech encoder, The vector conversion is as follows:

[0084]

[0085]

[0086] in, Represents a concatenation operation of parameters.

[0087] and Represents two consecutive layers of tokens in the language encoder model. Indicates the connected language encoder The transformation function of the layer, Indicates the language encoder Layer Group Tips, Indicates the language encoder Layer task prompts, and Corresponding to the language encoder Layer Group Tips Scaling Matrix, Layer task hint scaling matrix, and represent group source cues and task source cues, respectively;

[0088] Finally, through the projection function Will Projecting to the latent embedding space of the visual-language model to obtain the final language representation .

[0089] For visual input, we first perform patch embedding on the visual input. The visual encoder uses aligned visual images, infrared images, and depth images as joint inputs. The three visual inputs are connected in the channel dimension to form an input image in the format of ,here Indicates the resolution of the image, that is, the matrix height and length, Indicates the number of channels of the image. Here, since the visible light image, infrared image and depth image are connected, The input image is reorganized into a set of input 2D image blocks, and the composed image blocks can be expressed as , where the size of the image block is , the number of image blocks The image patches are then transformed into dimensional potential vector, complete block vectorization, calculated as follows:

[0090]

[0091] in, Represents different image blocks, represents the image vectorization operation, Represents image position vectorization operations Represents the dimensional space where the data is located;

[0092] Then combined with the learnable category label , input into the visual encoder.

[0093] See also Figure 2 Each layer of the visual encoder layer is composed of a multi-head self-attention module (MSA) and a multi-layer perceptron (MLP). At the beginning of each layer, layer normalization is performed ( ), a residual connection is used between MSA and MLP, and the multilayer perceptron contains two GELU nonlinear activation layers, which are stacked in sequence to visual encoder layer. layer, is passed to Layers are processed sequentially by the following transformers:

[0094]

[0095]

[0096]

[0097]

[0098] in, Indicates Class Token and image words The connection vector of yes and The intermediate variable of calculation, Represents the visual encoder The layer's group prompt, Represents the visual encoder Individual task prompts for each layer. They represent the visual encoder The layer's group prompt scaling matrix, task prompt scaling matrix, Respectively represent The group source hint and task source hint of the layer, represents the normalized result, and Indicates The category words and image words of the layer;

[0099] In addition, the individual task prompts of the visual encoder During the training process, the language encoder and the visual encoder are dynamically updated according to the prompts of the previous layer. The calculation method can be expressed as follows:

[0100]

[0101] in, is the balance weight, controlling the dynamic update, The function controls the output probability and is 1. Represents the visual encoder Layer individual task prompts, Represents the first Layer individual task prompts.

[0102] The visual encoder network structure is as follows Figure 2 As shown. After the encoding conversion of the visual encoder, the image obtained by the last encoder is marked as , here we use the image projection function Projecting it into the potential embedding space of the visual-language model, the calculation is expressed as follows:

[0103]

[0104] During the training process, the adjustment of language encoder parameters is constrained to prevent it from being overly fine-tuned for a specific task, which would lead to local overfitting in multi-task training and loss of model generalization performance.

[0105] The training of the entire model is completed by jointly optimizing the group shared prompt parameter loss function, the individual task prompt parameter loss function and the language encoder parameter adjustment loss function. That is, the loss function used in the training process of the present invention includes three main parts: the group shared prompt parameter loss function, the individual task prompt parameter loss function and the language encoder parameter adjustment loss function.

[0106] The group shared prompt parameter loss function focuses on the optimization of the group shared prompt parameters on the training results during the training process and can be expressed as

[0107]

[0108] The single task prompt parameter loss function measures the optimization of the exclusive prompt parameters for training during single task training, which can be expressed as:

[0109]

[0110] In the above two formulas Indicates the task The cross entropy loss function.

[0111] The language encoder parameter adjustment loss function is used to learn the feature similarity between the frozen pre-trained visual-language model and the prompts obtained after multimodal prompt collaborative learning. Here, cosine similarity is used to measure this similarity, which is expressed as follows

[0112]

[0113] in, represents the concatenation of the group-initial source prompt and the task-initial source prompt, and They are group-initial source cues and task-initial source cues, respectively;

[0114] The loss function of the entire network structure is finally expressed as

[0115]

[0116] The present invention is mainly aimed at multi-task learning of visual inspection. Common inspection tasks include but are not limited to equipment operation status detection, equipment appearance detection, equipment temperature detection, equipment humidity detection, smoke and dust / polluted gas detection, instrument panel / indicator light detection, switch status detection, fire point detection, line detection, foreign object / personnel intrusion detection, etc. There are similarities between some inspection tasks, but they are obviously different from other tasks, and the training data set that can be collected can only be a subset of the actual situation. In many cases, the abnormal situations that need to be detected cannot be reasonably trained with the existing data set. Therefore, the generalization of the visual-language model is guaranteed to be able to perform more effective detection for scenes that have not appeared. In order to verify the effectiveness of the multi-task learning method for visual inspection based on multimodal prompt collaboration proposed in the present invention, a variety of visual inspection data sets are used, and each data set is divided into two non-overlapping subsets: a training set and a validation set, to ensure that the scenes in the validation set do not appear in the training set, and in this way evaluate the generalizability of the proposed method. In addition, comparative experiments were conducted with several popular multi-task learning methods (C-Adapter, VPT-MT, BitFit-MT, CoOp-MT, CoCoOp-MT, MaPle-MT). The experimental results are shown below.

[0117] Table 1 Comparative experiment of multi-task learning methods for visual inspection based on multimodal prompt collaboration

[0118]

[0119] To sum up, the present invention aims at the multi-task learning tasks in visual inspection, and aims at the shortcomings of single visual source and multi-visual source methods such as susceptibility to interference and difficulty in fusion and trust. A multimodal prompt collaborative learning framework is adopted to solve the above problems. In view of the particularity of visual inspection tasks, an isomorphic fusion multimodal prompt collaborative learning framework is proposed. Adjustments are made to language prompts and visual prompts at the same time to enhance the correlation between the two, and the adjustment degree of the language encoder is constrained to ensure that the model does not only optimize the language prompt content, but also the detection performance decreases for the non-appearing content. Ultimately, the adjustment effect of the visual inspection multi-task model is improved to ensure the pertinence and generalization of the visual inspection multi-task learning method.

[0120] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0121] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A multi-task learning method for visual inspection based on multimodal prompt collaboration, characterized in that: A multi-task learning network with multi-modal prompts is used as a visual-language model to implement visual inspection multi-task learning. The multi-modal prompt collaborative multi-task learning network includes a language prompt encoder and a visual encoder. The language prompt encoder includes a parallel frozen language encoder and a language encoder. Freezing the speech encoder to ensure that the adjustment of the speech encoder is not greater than a threshold limit; The language encoder and the visual encoder use a coupling function to calculate the cosine similarity of their features to measure the similarity of the two tasks, so as to maximize the consistency of the visual and language representations of the task group with high total similarity and realize the joint training of multi-task learning of visual inspection. The method comprises the following steps: Step 1: Set the parameters of the multi-modal prompt collaborative multi-task learning network; Step 2: Freeze the pre-trained visual-language model, given the initial shared source cues for all tasks, calculate the gradient cosine similarity of the language encoder and the visual encoder for the shared cue parameters of the two tasks through the coupling function, measure the similarity of the two tasks, and group the tasks in the visual inspection multi-task training dataset; Step 3: For each group of tasks, provide group-initial source prompts and task-initial source prompts, combine with the scaling matrix, and generate prompts for the language encoder and visual encoder; Step 4: Based on the cues from the language encoder and the visual encoder, the training set of each task is divided into multiple batches for training, where each training batch contains several language-visual cue pairs; The loss function used in step 4 training is: in, They are group shared cue parameter loss function, single task cue parameter loss function, and language encoder parameter adjustment loss function; represents the concatenation of the group cue and the task cue in the language encoder, represents the concatenation of the group-initial source prompt and the task-initial source prompt, and They are group-initial source cues and task-initial source cues, respectively; They correspond to the language encoder group prompt scaling matrix, the visual encoder group prompt scaling matrix, the language encoder task prompt scaling matrix, and the visual encoder task prompt scaling matrix, respectively. is the penalty weight, which controls the magnitude of the weight adjustment after the language encoder is adjusted during training. Indicates the number of language encoder layers.

2. The multi-task learning method for visual inspection based on multi-modal prompt collaboration according to claim 1 is characterized in that: The step 2 calculates the gradient cosine similarity of the shared prompt parameters of the language encoder and the visual encoder through the coupling function, which is as follows: in, represents the gradient cosine similarity, Represents two different tasks for language encoder and visual encoder, Indicates the initial shared source prompt; represents the gradient of the initial shared source cue; represents the cross entropy loss function.

3. The visual inspection multi-task learning method based on multimodal prompt collaboration according to claim 1 is characterized in that: The prompts of the language encoder and the visual encoder in step 3 are generated as follows: in, , The language and visual cues for a single task are for the language encoder and the visual encoder, respectively; , are the initial language source cue and visual source cue for the language encoder and visual encoder, respectively; represents the Kronecker product operation; and They are group-initial source cues and task-initial source cues, respectively; They correspond respectively to the language encoder group prompt scaling matrix, the visual encoder group prompt scaling matrix, the language encoder group task prompt scaling matrix, and the visual encoder group task prompt scaling matrix.

4. The visual inspection multi-task learning method based on multimodal prompt collaboration according to claim 1 is characterized in that: In step 4, for the language prompt, the language prompt is first projected into the word vector space after lexical distraction, and then the word vector space is obtained. , represents the length of the word vector; Encode each language prompt into length, and then through a vector of length The language encoder is transformed, and finally we get The speech encoder output ; At each transformation layer of the speech encoder, The vector conversion is as follows: in, Indicates the concatenation operation of parameters; and Represents two consecutive layers of word units in the language encoder model, Indicates the connected language encoder The transformation function of the layer, Indicates the language encoder Layer Group Tips, Indicates the language encoder Layer task prompts, and Corresponding to the language encoder Layer group prompt scaling matrix, task prompt scaling matrix, and represent group source prompts and task source prompts respectively; Finally, through the projection function The language encoder Layer Output Projecting to the latent embedding space of the visual-language model to obtain the final language representation .

5. The visual inspection multi-task learning method based on multimodal prompt collaboration according to claim 1 is characterized in that: In step 4, for the visual input, firstly, block vectorization of the visual input is performed, wherein the aligned visual image, infrared image and depth image are used as the joint visual input image, and the three visual inputs are connected in the channel dimension to form an input image format of ,in represents the matrix height and length, Indicates the number of channels of the image; the input image is reorganized into a set of input 2D image blocks, the image blocks , the size of the image block is , the number of image blocks , and then use a trainable linear projection function to transform the image patch to dimensional potential vector, complete block vectorization, calculated as follows: in, Represents different image blocks, represents the image vectorization operation, Represents image position vectorization operations Represents the dimensional space where the data is located; Then with the learnable category labels Combined into , input into the visual encoder.

6. The visual inspection multi-task learning method based on multimodal prompt collaboration according to claim 5 is characterized in that: Each layer of the visual encoder is composed of a multi-head self-attention module MSA and a multi-layer perceptron MLP. Layer normalization is performed at the beginning of each layer. A residual connection is used between MSA and MLP. The multi-layer perceptron MLP contains two GELU nonlinear activation layers, which are superimposed in sequence to visual encoder layer; for the layer, is passed to Layers are processed sequentially by the following transformers: in, Indicates Class Token and image words The connection vector of yes and The intermediate variable of calculation, Represents the visual encoder The layer's group prompt, Represents the visual encoder Individual task prompts for each layer. They represent the visual encoder The layer's group prompt scaling matrix, task prompt scaling matrix, Respectively represent The group source hint and task source hint of the layer, represents the normalized result, and Indicates The category words and image words of the layer; , They are self-attention, normalization, and multi-layer perception functions respectively.

7. The visual inspection multi-task learning method based on multimodal prompt collaboration according to claim 6 is characterized in that: Individual task visual cues for the visual encoder During the training process, the language encoder and the visual encoder are dynamically updated according to the prompts of the previous layer. The calculation method is expressed as follows: in, is the balance weight, controlling the dynamic update, The function controls the output probability and is 1. Represents the visual encoder Layer individual task prompts, Represents the first Layer individual task prompts.

8. The visual inspection multi-task learning method based on multimodal prompt collaboration according to claim 7 is characterized in that: After the encoding conversion of the visual encoder, the image obtained by the last encoder layer is marked as , using the image projection function Project it into the latent embedding space of the visual-language model to obtain the final visual representation: .

Images