Visual language model training method, device, medium and computer program product

Abstract

The present invention relates to the field of artificial intelligence visual processing technology, and specifically discloses a visual language model training method, device, medium and computer program product. By establishing a hierarchical visual feature library of global visual features, foreground visual features and aggregated visual features for each example image in an image data set, a visual prompter of the visual language model is used to perform hierarchical screening when training a visual language model to improve the efficiency and quality of querying example images. The model training loss value is calculated using the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result to update the parameters of the visual perception model, thereby achieving global optimization of the visual language model. Compared with the local module adaptation method used in the traditional solution, it can avoid falling into the local optimum, thereby further improving the image processing quality.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence visual processing technology, and in particular to a visual language model training method, device, medium and computer program product. Background Art

[0002] The Visual Language Model (VLM) demonstrates excellent performance in a variety of visual and language tasks by learning the relationship between images and text, such as image classification, image generation, image retrieval, and natural language description of images. In the calculation process of the visual language model, we can query the image dataset for example images that are highly similar to the input image, and generate prompts from the input image, example image, and the image processing results of the example image and input them into the visual language model, so that the visual language model can output the image processing results of the query image without retraining or only with minimal fine-tuning. The query performance of the query example image determines the performance of the visual language model in performing image processing tasks.

[0003] How to improve the performance of visual language models in performing image processing tasks is a technical problem that technicians in this field need to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a visual language model training method, device, medium and computer program product for improving the performance of the visual language model in performing image processing tasks.

[0005] In order to solve the above technical problems, the present invention provides a visual language model training method, comprising:

[0006] Acquire a query image and an image data set, extract global visual features and foreground visual features of example images in the image data set, and aggregate similar foreground visual features to obtain aggregated visual features;

[0007] Extracting query visual features of the query image using a visual prompter in a visual language model, and performing step-by-step similarity screening in the order of the aggregate visual features, the foreground visual features, and the global visual features according to the query visual features to obtain an associated example image of the query image;

[0008] Inputting the query image, the associated example image and the associated example image processing result into the visual language model, and outputting the image processing result of the query image;

[0009] The model training loss value is calculated based on the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result;

[0010] The model parameters of the visual language model are updated using the model training loss value until a model training end condition is reached, thereby obtaining the trained visual language model.

[0011] On the one hand, according to the query visual feature, a step-by-step similarity screening is performed in the order of the aggregate visual feature, the foreground visual feature, and the global visual feature to obtain an associated example image of the query image, including:

[0012] Performing similarity screening on the query visual feature and each of the aggregated visual features to obtain a candidate aggregated visual feature associated with the query visual feature;

[0013] Performing similarity screening on the query visual feature and the foreground visual feature corresponding to the candidate aggregated visual feature to obtain a candidate foreground visual feature associated with the query visual feature;

[0014] Performing similarity screening on the query visual feature and the global visual feature corresponding to the candidate foreground visual feature to obtain an associated global visual feature associated with the query visual feature;

[0015] The example image corresponding to the associated global visual feature is used as the associated example image.

[0016] On the other hand, similarity screening is performed between the query visual feature and each of the aggregated visual features to obtain a candidate aggregated visual feature associated with the query visual feature, including:

[0017] Establishing semantic associations between each of the aggregated visual features and the query visual feature, respectively, and calculating a first probability value of the aggregated visual feature appearing in the query visual feature;

[0018] The aggregated visual feature whose first likelihood value is greater than a first threshold is used as the candidate aggregated visual feature.

[0019] On the other hand, establishing semantic associations between each of the aggregated visual features and the query visual feature, and calculating a first possibility value of the aggregated visual feature appearing in the query visual feature, comprises:

[0020] Using a cross attention mechanism to construct first semantic association information between the aggregated visual feature and the query visual feature;

[0021] The first likelihood value is predicted based on the first semantic association information and the number of types of the aggregated visual features in the image data set.

[0022] On the other hand, similarity screening is performed between the query visual feature and the foreground visual feature corresponding to the candidate aggregate visual feature to obtain a candidate foreground visual feature associated with the query visual feature, including:

[0023] According to the foreground type label corresponding to the candidate aggregated visual feature, query and obtain the corresponding first candidate foreground visual feature;

[0024] Establishing semantic associations between each of the first candidate foreground visual features and the query visual feature, respectively, and calculating a second probability value of the first candidate foreground visual feature appearing in the query visual feature;

[0025] The first candidate foreground visual feature whose second likelihood value is greater than a second threshold is used as the candidate foreground visual feature.

[0026] On the other hand, establishing semantic associations between each of the first candidate foreground visual features and the query visual feature, and calculating a second probability value of the first candidate foreground visual feature appearing in the query visual feature, includes:

[0027] Using a cross-attention mechanism to construct second semantic association information between the first candidate foreground visual feature and the query visual feature;

[0028] The second likelihood value is predicted based on the second semantic association information and the number of types of the aggregated visual features in the image data set.

[0029] On the other hand, similarity screening is performed between the query visual feature and the global visual feature corresponding to the candidate foreground visual feature to obtain an associated global visual feature associated with the query visual feature, including:

[0030] Obtaining the global visual feature corresponding to the candidate foreground visual feature, recorded as the first global visual feature;

[0031] Establishing semantic associations between each of the first global visual features and the query visual feature respectively, and calculating similarity values ​​between each of the first global visual features and the query visual feature;

[0032] According to the descending order of the similarity values, a first first number of the first global visual features are adopted as the associated global visual features.

[0033] On the other hand, the step of calculating the similarity loss value between the associated example image and the query image includes:

[0034] A loss value between the candidate aggregated visual feature and the true label of the query image is calculated as the similarity loss value.

[0035] On the other hand, the step of calculating the similarity loss value between the associated example image and the query image includes:

[0036] Calculating the loss between the candidate aggregated visual feature and the true label of the query image to obtain a first similarity loss value;

[0037] Calculating the loss between the candidate foreground visual feature and the true label of the query image to obtain a second similarity loss value;

[0038] Calculating the loss between the associated global visual feature and the true label of the query image to obtain a third similarity loss value;

[0039] The similarity loss value is calculated according to the first similarity loss value, the second similarity loss value and the third similarity loss value.

[0040] On the other hand, the query image, the associated example image, and the associated example image processing result are input into the visual language model, and the image processing result of the query image is output, including:

[0041] According to the similarity with the query image from large to small, taking the first first number of the associated example images and the associated example image processing results of the associated example images as positive sample pairs;

[0042] According to the similarity with the query image from small to large, the first number of example images and the example image processing results of the example images are used as negative sample pairs;

[0043] The query image, the positive sample pair and the negative sample pair are input into the visual language model to obtain a positive sample image processing result based on the positive sample pair for the query image and a negative sample image processing result based on the negative sample pair for the query image.

[0044] On the other hand, the calculation steps of the processing loss value of the image processing result are:

[0045] Calculate the contrast loss value according to the positive sample image processing result, the negative sample image processing result and the label image processing result corresponding to the query image;

[0046] The contrast loss value is used as the processing loss value.

[0047] In order to solve the above technical problems, the present invention also provides an image processing method, comprising:

[0048] Acquire a query image and an image data set, extract global visual features and foreground visual features of example images in the image data set, and aggregate similar foreground visual features to obtain aggregated visual features;

[0049] Extracting query visual features of the query image using a visual prompter in a visual language model, and performing step-by-step similarity screening in the order of the aggregate visual features, the foreground visual features, and the global visual features according to the query visual features to obtain an associated example image of the query image;

[0050] Inputting the query image, the associated example image and the associated example image processing result into the visual language model, and outputting the image processing result of the query image;

[0051] The model training loss value is calculated based on the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result;

[0052] Using the model training loss value to update the model parameters of the visual language model until the model training end condition is reached, thereby obtaining the trained visual language model;

[0053] The input image is input into the trained visual language model to obtain a target example image associated with the input image from the image data set, and then the input image, the target example image and the example image processing result of the target example image are used to generate prompts for visual language calculation to obtain an image processing result of the input image.

[0054] In order to solve the above technical problems, the present invention further provides an electronic device, comprising:

[0055] Memory for storing computer programs;

[0056] A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the visual language model training method or the image processing method as described in any one of the above items are implemented.

[0057] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the visual language model training method or the image processing method as described in any one of the above items are implemented.

[0058] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the visual language model training method or the image processing method as described in any one of the above items.

[0059] The visual language model training method provided by the present invention has the beneficial effect of establishing a hierarchical visual feature library of global visual features, foreground visual features, and aggregated visual features for each example image in the image data set. When training the visual language model, the visual prompter of the visual language model is used to perform step-by-step similarity screening in the order from aggregated visual features, foreground visual features to global visual features to obtain the associated example image of the query image, so that it is not necessary to compare the query visual features with the global visual features of all example images, thereby improving the efficiency of obtaining the associated example images, and enabling the model to understand the relationship and difference between different types of features through this hierarchical relationship, thereby improving the closeness between the obtained example image and the query image. The model training loss value is calculated using the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result to update the parameters of the visual perception model, thereby realizing the global optimization of the visual language model. Compared with the local module adaptation method used in the traditional solution, it can avoid falling into the local optimum, thereby further improving the image processing quality.

[0060] The image processing method, electronic device, non-volatile storage medium and computer program product provided by the present invention have the above-mentioned beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 A flowchart of a visual language model training method provided by an embodiment of the present invention;

[0063] Figure 2 A calculation flow chart of a visual prompter provided by an embodiment of the present invention;

[0064] Figure 3 A flowchart of another visual language model training method provided by an embodiment of the present invention;

[0065] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The core of the present invention is to provide a visual language model training method, device, medium and computer program product for improving the performance of the visual language model in performing image processing tasks.

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] To facilitate understanding of the technical solution provided by the embodiment of the present invention, some key terms used in the embodiment of the present invention are first explained here.

[0069] Prompt Learning was first proposed in the field of Natural Language Processing (NLP). It gives language models the ability to directly model raw text. The core idea is to guide the model to perform specific tasks by constructing and using "prompts", that is, inputs in a specific format or template. This improves the model's performance in specific tasks without changing the original model parameters. This method can not only activate and apply the rich knowledge contained in the model to solve complex natural language processing problems, but also enhance the interpretability of the model and allow the model to quickly adapt to new tasks or new fields.

[0070] Visual language models have developed rapidly in recent years, mainly benefiting from the progress of deep learning technology and the availability of large-scale datasets. These models have demonstrated excellent performance in a variety of visual and language tasks, such as image classification, image generation, image retrieval, and natural language description of images by learning the association between images and text. Although these models have shown amazing capabilities, there are still many challenges in the adaptation of downstream tasks, such as the huge computing resources required for fine-tuning, some models only provide an application programming interface (API), and poor interpretability.

[0071] Visual In-Context Learning based on visual language models is an emerging research field, which is to complete the adaptation of new tasks without retraining the model or with only minimal fine-tuning. Specifically, only a query image (query) and an in-context pair are required to enable the visual language model to complete the processing of the query image. Among them, the context pair includes an example image and the processing result of the example image, and the example image is retrieved from an image dataset. It can be understood that the closer the example image is to the query image, the better the quality of the image processing result of the query image by the visual language model based on the example image and the image processing result of the example image. How to select better quality example images that are closer to the query image based on a pre-trained visual language model and a limited image dataset, so as to improve the performance of the visual language model in processing specific image processing tasks, is a technical problem that needs to be solved in this technical field.

[0072] In the related technology, the context learning based on the visual language model fine-tunes the visual feature extraction model, that is, the downstream task dataset is used to fine-tune the parameters of the visual feature model to adapt to the downstream task. The query image and the entire image visual information of all images in the dataset are extracted separately, and the similarity between the query image and all images in the dataset is calculated for screening, resulting in low retrieval efficiency and poor retrieval quality. In the entire process, the visual language model is not combined for global optimization, but some sub-modules are fine-tuned independently, resulting in sub-optimization problems.

[0073] To this end, the present invention provides a visual language model training scheme and an image processing scheme. By establishing a hierarchical visual feature library of global visual features, foreground visual features, and aggregated visual features for each sample image in the image data set, when training the visual language model, the visual prompter of the visual language model is used to perform step-by-step similarity screening in the order from aggregated visual features, foreground visual features to global visual features to obtain an associated sample image of the query image, so that it is not necessary to compare the query visual features with the global visual features of all sample images, thereby improving the efficiency of obtaining associated sample images, and enabling the model to understand the relationship and difference between different types of features through this hierarchical relationship, thereby improving the degree of proximity between the obtained sample image and the query image. The model training loss value is calculated using the similarity loss value between the associated sample image and the query image and the processing loss value of the image processing result to update the parameters of the visual perception model, thereby achieving global optimization of the visual language model. Compared with the local module adaptation method used in the traditional scheme, it can avoid falling into the local optimum, thereby further improving the image processing quality.

[0074] The purpose of an embodiment of the present invention is to train a visual language model and to use the trained visual language model to perform image processing tasks. The visual language model is an artificial intelligence model that combines computer vision and natural language processing. The training of the visual language model includes two stages, namely the pre-training stage and the fine-tuning stage. In the pre-training stage, it is generally based on an image data set, and for a specific model training task, a large-scale neural network algorithm structure is trained to learn and implement it. The large-scale neural network algorithm structure and parameters obtained in the end are the visual language pre-training model. In the fine-tuning stage, small-scale training is performed for specific task objectives (downstream tasks) and task data (downstream data) to achieve slight adjustments to the parameters of the pre-training model, and finally a model adapted to specific tasks and data is obtained. The neural network algorithm structure for visual language model training can be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), etc., or a model built with an attention network, such as a transformer model (Transformer), a bidirectional encoder representation from transformers (BERT), etc., which is not limited in the embodiments of the present invention. An attention network refers to a network model that uses an attention mechanism for training. The model extracts more important feature information from the input sequence by assigning different weights to each part of the input sequence, so that the model ultimately obtains a more accurate output.

[0075] The visual language model training method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0076] Figure 1 A flowchart of a visual language model training method provided by an embodiment of the present invention.

[0077] like Figure 1 As shown, the visual language model training method provided by the embodiment of the present invention includes:

[0078] S101: obtaining a query image and an image dataset, extracting global visual features and foreground visual features of example images in the image dataset, and aggregating similar foreground visual features to obtain aggregated visual features;

[0079] S102: extracting query visual features of the query image using the visual prompter in the visual language model, and performing step-by-step similarity screening in the order of aggregate visual features, foreground visual features, and global visual features according to the query visual features to obtain associated example images of the query image;

[0080] S103: inputting the query image, the associated example images and the associated example image processing results into the visual language model, and outputting the image processing results of the query image;

[0081] S104: Calculate a model training loss value according to a similarity loss value between the associated example image and the query image and a processing loss value of the image processing result;

[0082] S105: Using the model training loss value, the model parameters of the visual language model are updated until the model training end condition is reached, thereby obtaining a trained visual language model.

[0083] In a specific implementation, the visual language model training method provided by an embodiment of the present invention can be implemented based on a single computing device or a computing system composed of multiple computing devices. The computing device used can be, but is not limited to, a graphics processing unit (GPU), a field programmable gate array device (FPGA), an application specific integrated circuit device (ASIC), and a processor distributed processing unit device (DPU), or a computing device using one or more of them as an accelerator, and other types of computing devices may also be used.

[0084] In an embodiment of the present invention, for S101, a visual feature library with a hierarchical structure is constructed based on example images in the image data set, and the hierarchical structure is composed of global image features of each example image, foreground image features of each example image, and aggregated visual features obtained by aggregating similar foreground visual features.

[0085] In an embodiment of the present invention, a visual feature extraction model is used to extract query visual features of a query image and example image features of an example image. The visual feature extraction model may use an existing visual feature extraction model, such as a Vision Transformer (ViT) model, a Contrastive Language-Image Pretraining (CLIP) model, and the like.

[0086] For S102, a visual cue of the visual language model is constructed, and the visual cue is used to compare the similarity with the query visual features of the query image step by step in the order from aggregated visual features, foreground visual features to global visual features, so as to screen out associated example images that are highly similar to the query image step by step.

[0087] For S103, the associated example image and the associated example image processing result are used as context pairs associated with the query image, and the query image, the context pair and the task goal of the image processing task generate prompts (Prompt) and input them into the visual language model for visual language calculation, and the image processing result of the query image is output.

[0088] In the embodiment of the present invention, the query image may be an independent image or a video frame in a video. The image processing task may include but is not limited to one or a combination of image segmentation tasks, target recognition tasks, image classification tasks, image generation tasks, image retrieval tasks, and natural language description image tasks.

[0089] For S104, the image query task of the visual prompter and the visual language model are jointly optimized, and the similarity loss value between the associated example image and the query image, as well as the processing loss value between the image processing result of the visual language model on the query image and the true label of the query image are calculated respectively, and the model training loss value is obtained by comprehensive calculation.

[0090] For S105, the model parameters of the visual language model are updated using the model training loss value, which not only improves the quality of the query example image, but also fully explores the potential knowledge of the visual language model, avoids the local optimal problem caused by not incorporating the visual language model into the learning process, improves the performance of the visual language model in contextual learning, and further improves the applicability of the visual language model.

[0091] In an embodiment of the present invention, the model training termination condition may be that a preset number of iterative training times is reached, or the model training loss value of the visual language model is reduced to an error threshold.

[0092] The visual language model training method provided by the embodiment of the present invention establishes a hierarchical visual feature library of global visual features, foreground visual features, and aggregated visual features for each example image in the image data set. When training the visual language model, the visual prompter of the visual language model is used to perform step-by-step similarity screening in the order from aggregated visual features, foreground visual features to global visual features to obtain the associated example image of the query image, so that there is no need to compare the query visual features with the global visual features of all example images, which improves the efficiency of obtaining the associated example images, and enables the model to understand the relationship and difference between different types of features through this hierarchical relationship, thereby improving the closeness between the obtained example image and the query image. The model training loss value is calculated using the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result to update the parameters of the visual perception model, thereby realizing the global optimization of the visual language model. Compared with the local module adaptation method used in the traditional solution, it can avoid falling into the local optimum, thereby further improving the image processing quality.

[0093] Based on the above embodiment, the embodiment of the present invention further illustrates the calculation steps of the visual prompter.

[0094] Figure 2 A calculation flow chart of a visual prompter provided by an embodiment of the present invention.

[0095] Since the content in the image is composed of multiple elements, such as people, cats, bicycles, etc., the embodiment of the present invention provides an example image screening strategy from local to overall. First, with the query visual features of the query image as the target, candidates with similar features are preliminarily screened out from the image data set; then, based on the preliminary screening results, the relationship between the elements and their configuration in the overall scene are analyzed, and examples that highly match the query visual features and contextual content are further selected. This method not only improves the quality and efficiency of screening, but also ensures that the finally selected associated example images have strong visual and semantic consistency with the query image.

[0096] In an embodiment of the present invention, constructing a visual feature library with a hierarchical structure may further include: extracting global visual features of each example image in an image data set; extracting foreground visual features of each example image in an image data set and constructing a visual feature library with a hierarchical structure; and building a visual prompter to filter associated example images of a query image from the image data set.

[0097] The extraction of the global visual features of each example image in the image data set may include: using a first visual feature extraction model to extract the global visual features of all example images in the image data set, denoted as .

[0098] Extracting foreground visual features of each example image in the image data set and constructing a visual feature library with a hierarchical structure may include: extracting foreground visual features of all example images in the image data set, constructing foreground visual features with a hierarchical structure, and obtaining a visual feature library.

[0099] The step of extracting the foreground visual features of all the example images in the image dataset may include: using a second visual feature extraction model to extract the foreground visual features of all the example images in the image dataset. Taking the image processing task as an image segmentation task as an example, the image dataset provides real segmentation results and labels. Here, the segmentation results are used as masks, and the second visual feature extraction model is used to extract the foreground visual features of all the example images in the image dataset, which is recorded as , Representation example image The corresponding foreground visual features. The second visual feature extraction model may adopt the same visual feature extraction model as the first visual feature extraction model.

[0100] In order to improve the accuracy and timeliness of the visual prompter in screening the example images, in the embodiment of the present invention, based on the extraction of foreground visual features, similar foreground visual features are aggregated to obtain aggregated visual features. Here, similar foreground visual features may refer to features with the same label in the image dataset.

[0101] In the embodiment of the present invention, a foreground visual feature with a hierarchical structure is constructed to obtain a visual feature library. First, all foreground visual features are extracted and integrated. The integration can be performed using the real labels provided by the image dataset or the similarity in the feature space. Here, the real labels are used. That is, all foreground visual features in the image dataset are extracted and integrated. Based on the type of masked true label, the foreground visual features of the same type are integrated, and the result is recorded as ,in, are all foreground visual features corresponding to type i, specifically ;in, is the image index, and .

[0102] Aggregate the same type of foreground visual features. Since the same type of foreground visual features have similar visual information, in order to improve the efficiency of the visual prompter in screening sample images, the mean pooling calculation method can be used here to aggregate the same type of foreground visual features, that is, ,in, represents the set of all aggregated visual features, , It is the aggregation result of all foreground visual features Feat_i in type i.

[0103] Constructing a visual feature library with hierarchical relationships As shown in Table 1, this table gives a simple example of a visual feature library with a hierarchical relationship, where the first column label is the corresponding foreground type, such as cat, dog, etc., the second column is the aggregated visual feature, which means that the foreground visual features of the same type in the image dataset are obtained by average pooling calculation, the third column is the foreground visual features of the same type, and the fourth column is the original example image index containing such foreground visual information.

[0104] Table 1

[0105]

[0106] Based on the above visual feature library Build a visual prompter. In order to filter out associated example images that are closer to the query image from the image data set, the query visual features of the query image are respectively interacted with the coarse foreground visual information, the fine foreground visual information, and the global visual information for step-by-step screening and fusion to improve the similarity of the screening results. Then, in S102, according to the query visual features, step-by-step similarity screening is performed in the order from aggregate visual features, foreground visual features to global visual features to obtain associated example images of the query image, which may include: similarity screening the query visual features with each aggregate visual feature to obtain a candidate aggregate visual feature associated with the query visual feature; similarity screening the query visual features with the foreground visual features corresponding to the candidate aggregate visual features to obtain a candidate foreground visual feature associated with the query visual feature; similarity screening the query visual features with the global visual features corresponding to the candidate foreground visual features to obtain an associated global visual feature associated with the query visual feature; and taking the example image corresponding to the associated global visual feature as the associated example image.

[0107] In the specific implementation, Figure 2 As shown, first obtain all aggregated visual features And a visual feature extraction model is used to extract query visual features of the query image.

[0108] In some optional implementations of the embodiments of the present invention, the foreground visual features associated with the query image are dynamically captured by predicting the foreground visual features associated with the query image, so as to quickly screen out the visual features associated with the query image. The query visual features are then screened for similarity with each aggregated visual feature to obtain a candidate aggregated visual feature associated with the query visual feature, which may include: establishing semantic associations between each aggregated visual feature and the query visual feature, calculating a first probability value of the aggregated visual feature appearing in the query visual feature; and taking the aggregated visual feature whose first probability value is greater than a first threshold as a candidate aggregated visual feature.

[0109] Among them, establishing semantic associations between each aggregated visual feature and the query visual feature respectively, and calculating a first possibility value of the aggregated visual feature appearing in the query visual feature can include: using a cross-attention mechanism to construct first semantic association information between the aggregated visual feature and the query visual feature; and predicting the first possibility value based on the first semantic association information and the number of types of aggregated visual features in the image data set.

[0110] Specifically, a multi-headed cross-attention mechanism can be used to construct the semantic association between the query image visual features and the aggregated visual features, forcing this semantic association to model the interaction in parallel in different representation subspaces, i.e. , Indicates the prediction result, i.e. the first possibility value, is the mapping function, and For two inputs, the specific steps are as follows:

[0111] Linearly map the query visual features and the aggregated visual features to feature spaces with the same dimension: , ,in, for Mapping results, for Mapping results, and is the number and dimension of the attention heads.

[0112] The multi-head attention mechanism is used to establish the semantic information association between the query visual features and the aggregated visual features. The query visual features of the query image are used as the query statement (query), and the semantic information of the aggregated visual features are used as the key (key) and value (value). The semantic association information between the two is established in different subspaces: .

[0113] Predict the likelihood of the aggregated visual feature in the query visual feature, i.e. , where Sigmod is the activation function, Linear is the fully connected mapping function, and attention is the output result of the multi-head attention mechanism. is the number of types of aggregated visual features in the image dataset.

[0114] At this point, the loss between the above prediction result, i.e., the first likelihood value, and the true label of the query visual feature can be calculated as the similarity loss value. This is a multi-label classification problem, and binary cross entropy can be used as the loss function here, i.e. , where BCE() represents the binary cross entropy loss function, the input is the prediction result pred of the multi-head attention mechanism, and query_lable is the true label of the element in the query visual feature.

[0115] Then, filter the prediction results. According to the above steps, the first probability value of each aggregated visual feature appearing in the query visual feature is calculated. In order to further improve the accuracy of the final sample image screening, a large amount of useless information is filtered and the prediction results are filtered, that is, Here, thr is the first threshold value, which can be set to 0.6.

[0116] In an embodiment of the present invention, a query visual feature and a foreground visual feature corresponding to a candidate aggregate visual feature are similarly screened to obtain a candidate foreground visual feature associated with the query visual feature, which may include: querying and obtaining a corresponding first candidate foreground visual feature based on a foreground type label corresponding to the candidate aggregate visual feature; establishing a semantic association between each first candidate foreground visual feature and the query visual feature, and calculating a second possibility value of the first candidate foreground visual feature appearing in the query visual feature; and using the first candidate foreground visual feature whose second possibility value is greater than a second threshold as a candidate foreground visual feature.

[0117] Among them, establishing semantic associations between each first candidate foreground visual feature and the query visual feature respectively, and calculating the second possibility value of the first candidate foreground visual feature appearing in the query visual feature can include: using a cross-attention mechanism to construct second semantic association information between the first candidate foreground visual feature and the query visual feature; and predicting the second possibility value based on the second semantic association information and the number of types of aggregated visual features in the image data set.

[0118] Specifically, based on the prediction results of the previous step and visual feature library , get the first foreground visual feature of the corresponding type ,in , is the type of candidate aggregated visual features.

[0119] From the type-related first foreground visual features Further accurately screening the candidate foreground visual features corresponding to the query visual feature may include: traversing the first foreground visual feature All foreground visual features in , let the label of the current foreground visual feature be , the corresponding first candidate foreground visual feature is . Construct query visual features and first candidate foreground visual features The semantic association of , For type tags The corresponding first candidate foreground visual feature The prediction results here are is a mapping function with a cross-attention mechanism, which is the same as the above operation.

[0120] Then, filter the prediction results. According to the above steps, calculate the second probability value of each aggregated visual feature appearing in the query visual feature, filter the prediction results, and set the second threshold to 0.7, that is, ,in The second threshold is 0.7.

[0121] Repeat the above steps until completed Traverse, that is .

[0122] Based on candidate foreground visual features and the global visual features of all example images in the image dataset , further accurately obtain the associated global visual features of the associated example images corresponding to the query visual features. In an embodiment of the present invention, similarity screening is performed on the query visual features and the global visual features corresponding to the candidate foreground visual features to obtain the associated global visual features associated with the query visual features, which may include: obtaining the global visual features corresponding to the candidate foreground visual features, recorded as the first global visual features; establishing semantic associations between each first global visual feature and the query visual feature, respectively, and calculating the similarity values ​​between each first global visual feature and the query visual feature; and using the first first number of first global visual features as the associated global visual features in descending order of similarity values.

[0123] Specifically, we first traverse the candidate foreground visual features All elements in , let the current element be , i is the corresponding index, Based on the candidate foreground visual features and visual feature library , from all global visual features Obtain the corresponding visual features of the entire image .

[0124] Constructing query image features and The association, that is , is the visual feature of the entire image corresponding to type label i The prediction results here are is a mapping function with a cross-attention mechanism, which is the same as the above operation.

[0125] Repeat the above steps until all All elements in , and sum up all the results, that is, .

[0126] Based on the query image and global visual features The prediction results , obtain the corresponding example image as the associated example image, and obtain the associated example image processing result of the associated example image to complete the selection of the in-context pair. The process may include: obtaining the query image and the global visual features The prediction results , further fused using the fully connected layer, and finally using the softamax operation to obtain the screened Relative to the similarity of the query visual features, that is, From the above results Select top-k and find the associated example image corresponding to the threshold and the associated example image processing result as the final filtered context pair. k can be selected as 3.

[0127] As introduced above, the loss of the prediction result predicted in the first step can be used as the similarity loss value constituting the model training loss value, that is, the calculation steps of the similarity loss value between the associated example image and the query image may include: calculating the loss value between the candidate aggregated visual features and the true label of the query image as the similarity loss value. The specific calculation process is as described above. Records of.

[0128] In some other optional implementations of the embodiments of the present invention, the step of calculating the similarity loss value between the associated example image and the query image may also include: calculating the loss between the candidate aggregated visual features and the true label of the query image to obtain a first similarity loss value; calculating the loss between the candidate foreground visual features and the true label of the query image to obtain a second similarity loss value; calculating the loss between the associated global visual features and the true label of the query image to obtain a third similarity loss value; and calculating the similarity loss value based on the first similarity loss value, the second similarity loss value and the third similarity loss value.

[0129] That is to say, the similarity loss value can be calculated by using the joint loss of each step prediction, where the calculation methods of the first similarity loss value, the second similarity loss value, and the third similarity loss value can all refer to Records of.

[0130] Further, calculating the query loss value according to the first similarity loss value, the second similarity loss value and the third similarity loss value may include: obtaining the similarity loss value by weighted calculation according to the first similarity loss value, the second similarity loss value and the third similarity loss value. The visual language model training method provided by the embodiment of the present invention may also include: updating the first self-learning weight of the first similarity loss value, the second self-learning weight of the second similarity loss value and the third self-learning weight of the third similarity loss value using the model training loss value.

[0131] Through the joint loss, the screening quality of the feature screening method from coarse to fine and from local to global formed by the hierarchical visual feature library constructed in the embodiment of the present invention is added to the joint optimization of the visual language model, thereby further improving the quality of the query-related example images.

[0132] The training method of the visual language model provided by the embodiment of the present invention, by constructing a visual feature library and a visual prompter with a hierarchical relationship, helps the model to establish a clear boundary between the foreground and the background through the hierarchical relationship, reduces confusion and erroneous feature attribution, thereby improving the accuracy of the overall prediction, so that the visual language model first screens out candidates from a large range, and then gradually narrows down to specific details. This strategy from coarse to fine significantly improves the speed and efficiency of retrieval. Hierarchical features provide the model with the possibility of feature fusion at different levels, which enables the model to select the most relevant feature set according to the needs of different tasks. At the same time, by focusing on the foreground elements in the image, the visual prompter avoids the interference of background information and reduces unnecessary information processing burden. Using a hierarchical retrieval strategy, the visual prompter effectively locates relevant information quickly from a large amount of data, thereby improving retrieval efficiency. The visual prompter can also flexibly adjust the visual features identified and extracted according to different application requirements to adapt to the changing task environment.

[0133] Based on the above embodiment, the embodiment of the present invention further illustrates the step of updating the model parameters of the visual language model.

[0134] Figure 3 A flowchart of another visual language model training method provided by an embodiment of the present invention.

[0135] In the above embodiment, the calculation method of the similarity loss value is introduced. In some optional implementations of the embodiments of the present invention, the processing loss value of the image processing result of the visual language model on the query image can be the processing loss value between the image processing result of the query image and the label image processing result of the query image.

[0136] In some other optional implementations of the embodiments of the present invention, in order to further improve the learning effect of the visual language model, based on the context pairs screened out by the visual prompter as positive sample pairs, corresponding negative sample pairs are constructed and jointly input into the visual language model to construct comparative learning.

[0137] Based on this, in S103, the query image, associated example images and the associated example image processing results are input into the visual language model, and the image processing results of the query image are output, which may include: according to the similarity with the query image from large to small, the first first number of associated example images and the associated example image processing results of the associated example images are used as positive sample pairs; according to the similarity with the query image from small to large, the first first number of example images and the example image processing results of the example images are used as negative sample pairs; the query image, the positive sample pair and the negative sample pair are input into the visual language model to obtain the positive sample image processing results of the query image based on the positive sample pair and the negative sample image processing results of the query image based on the negative sample pair.

[0138] like Figure 3 As shown, the visual prompter is used to filter out the top-k associated example images associated with the query image and the associated example image processing results as positive sample pairs according to the visual feature library, and a negative sample pair is constructed.

[0139] In an embodiment of the present invention, the method for constructing a negative sample pair may include: selecting the one with the worst prediction result according to the prediction result of the visual prompter. The example images and the example image processing results corresponding to the results; during the training process of the visual language model, randomly select different types of training samples in the same batch sample images and sample image processing results; remove the current training batch data and other training sets that do not belong to the current category from the image dataset, and randomly select Based on one or more of the above sources, k ( ) negative sample pairs.

[0140] like Figure 3As shown, data enhancement is performed on positive sample pairs and negative sample pairs respectively, thereby improving the generalization performance and robustness of the model. The data enhancement method may include: adding random perturbations to the sample image, such as adding noise to the original image using a uniform distribution with a mean of 1 and a variance of 40 to obtain a first sample image. Flipping the sample image horizontally to obtain a second sample image. Crop the sample image with the target as the center and randomly rotate it to obtain a third sample image. Through one or more of the above methods, sample images and image processing results for data enhancement of positive sample pairs and negative sample pairs are obtained. Through the first sample image, the robustness of the model can be enhanced. Through the second sample image, the model can be forced to better understand the object. Through the third sample image, background interference can be reduced, forcing the model to learn the features of the same object from different perspectives, thereby improving the generalization performance of the model.

[0141] Based on this, the steps for calculating the processing loss value of the image processing result can be: calculating the contrast loss value according to the positive sample image processing result, the negative sample image processing result and the label image processing result corresponding to the query image; and using the contrast loss value as the processing loss value.

[0142] In a specific implementation, the enhanced positive sample pairs and enhanced negative sample pairs in the above steps, as well as the query image, can be used to input the visual language model respectively to obtain the positive sample image processing results respectively. , negative sample image processing results .make and are the enhanced positive sample pairs and the enhanced negative sample pairs respectively. The prediction result of the query image is an empty set, which is recorded as , then the visual language calculation steps of the visual language model include:

[0143] according to The positive sample image processing results are calculated, where LVM represents the visual language model. To enhance the positive sample pair And the positive sample image processing result obtained by querying the image input visual language model;

[0144] according to The negative sample image processing results are calculated, where LVM represents the visual language model. To enhance the negative sample pair And the negative sample image processing results obtained by querying the image input visual language model.

[0145] Construct contrast loss. This is done by enhancing the distinguishability of the model's internal features and making the classes more similar and the classes more different. That is, the positive sample image processing results, the negative sample image processing results, and the label image processing results of the query image are used to construct the contrast loss. Construct contrast loss:

[0146] ;

[0147] in, Indicates the processing loss value; represents a set of query image samples; Indicates the number of query images used in the current training batch; Represents the image segmentation task performance metric function (Intersection over Union), which is used to evaluate the similarity between the predicted box (bounding box) and the true box (ground-truth); Represents logarithmic operation; Represents exponential operation.

[0148] The training method of the visual language model provided in the embodiment of the present invention utilizes a contrastive learning strategy to mine deeper knowledge and patterns from the visual language model and provide richer contextual information for complex visual tasks. By comparing different positive and negative samples, the visual language model can learn and refine more discriminative feature representations from a global perspective, which helps to understand the complex nature of the image and achieve comprehensive feature learning. The joint training strategy forces the model to recognize and distinguish various changes. Such a training mechanism enhances the model's generalization ability to new unknown data.

[0149] The above describes in detail various embodiments corresponding to the training method of the visual language model. On this basis, the present invention also discloses an image processing method, which may include:

[0150] Obtain a query image and an image dataset, extract global visual features and foreground visual features of example images in the image dataset, and aggregate similar foreground visual features to obtain aggregated visual features;

[0151] The query visual features of the query image are extracted by using the visual prompter in the visual language model, and the query visual features are screened in a step-by-step similarity order from the aggregate visual features, the foreground visual features to the global visual features to obtain the associated sample images of the query image;

[0152] Inputting the query image, the associated example image and the associated example image processing result into the visual language model, and outputting the image processing result of the query image;

[0153] The model training loss value is calculated based on the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result;

[0154] The model parameters of the visual language model are updated using the model training loss value until the model training end condition is reached to obtain the trained visual language model;

[0155] The input image is input into the trained visual language model, and after obtaining the target example image associated with the input image by querying the self-image dataset, the input image, the target example image and the example image processing result of the target example image are used to generate prompts for visual language calculation to obtain the image processing result of the input image.

[0156] For the specific implementation of the image processing method introduced in the embodiment of the present invention, please refer to the introduction of the steps of the training method of the visual language model.

[0157] It should be noted that in the embodiments of the training method and image processing method of each visual language model of the present invention, some of the steps or features can be ignored or not executed. The hardware or software functional modules divided for the convenience of explanation are not the only implementation form of the training method and image processing method of the visual language model provided in the embodiments of the present invention.

[0158] The above details various embodiments corresponding to the training method of the visual language model and the image processing method. On this basis, the present invention also discloses a visual language model training device, an image processing device, an electronic device, a non-volatile storage medium and a computer program product corresponding to the above method.

[0159] The visual language model training device provided by the embodiment of the present invention may include:

[0160] A visual feature library construction module is used to obtain a query image and an image data set, extract global visual features and foreground visual features of example images in the image data set, and aggregate similar foreground visual features to obtain aggregated visual features;

[0161] An example query module is used to extract query visual features of the query image using the visual prompter in the visual language model, and perform step-by-step similarity screening in the order of aggregate visual features, foreground visual features and global visual features according to the query visual features to obtain an associated example image of the query image;

[0162] The model training module is used to input the query image, the associated example image and the associated example image processing result into the visual language model, and output the image processing result of the query image; calculate the model training loss value according to the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result; use the model training loss value to update the model parameters of the visual language model until the model training end condition is met, and obtain the trained visual language model.

[0163] The image processing device provided by the embodiment of the present invention may include:

[0164] A visual feature library construction module is used to obtain a query image and an image data set, extract global visual features and foreground visual features of example images in the image data set, and aggregate similar foreground visual features to obtain aggregated visual features;

[0165] An example query module is used to extract query visual features of the query image using the visual prompter in the visual language model, and perform step-by-step similarity screening in the order of aggregate visual features, foreground visual features and global visual features according to the query visual features to obtain an associated example image of the query image;

[0166] The model training module is used to input the query image, the associated example image and the associated example image processing result into the visual language model, and output the image processing result of the query image; calculate the model training loss value according to the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result; use the model training loss value to update the model parameters of the visual language model until the model training end condition is met, and obtain the trained visual language model;

[0167] The image processing module is used to input the input image into the trained visual language model, and after obtaining the target example image associated with the input image from the image data set query, the input image, the target example image and the example image processing result of the target example image are used to generate prompts for visual language calculation to obtain the image processing result of the input image.

[0168] It should be noted that in each embodiment of the visual language model training device and the image processing device provided by the embodiments of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection between different units can be electrical, mechanical or other connection methods. Separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiments of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0169] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0170] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0171] like Figure 4As shown, the electronic device provided by the embodiment of the present invention includes: a memory 410, used to store a computer program 411; a processor 420, used to execute the computer program 411, and when the computer program 411 is executed by the processor 420, the steps of the training method and image processing method of the visual language model provided in any of the above embodiments are implemented.

[0172] Among them, the processor 420 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 420 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 420 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 420 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 420 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0173] The memory 410 may include one or more non-volatile storage media, which may be non-transitory. The memory 410 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 410 is at least used to store the following computer program 411, wherein the computer program 411, after being loaded and executed by the processor 420, can implement the training method of the visual language model and the related steps in the image processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 410 may also include an operating system 412 and data 413, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system 412 may be Windows or other types of operating systems. Data 413 may include, but is not limited to, the data involved in the above method.

[0174] In some embodiments, the electronic device may further include a display screen 430 , a power supply 440 , a communication interface 450 , an input / output interface 460 , a sensor 470 , and a communication bus 480 .

[0175] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure.

[0176] The electronic device provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the visual language model training method and the image processing method provided in the above embodiment, and the effect is the same as above.

[0177] An embodiment of the present invention provides a non-volatile storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the visual language model training method and the image processing method provided in any one of the above embodiments can be implemented.

[0178] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0179] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the training method and image processing method of the visual language model provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0180] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the visual language model training method and the image processing method provided in any one of the above embodiments.

[0181] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the training method and image processing method of the visual language model provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0182] The visual language model training method, device, medium and computer program product provided by the present invention are introduced in detail above. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0183] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A visual language model training method, characterized in that: include: Acquire a query image and an image data set, extract global visual features and foreground visual features of example images in the image data set, and aggregate similar foreground visual features to obtain aggregated visual features; Extracting query visual features of the query image using a visual prompter in a visual language model, and performing step-by-step similarity screening in the order of the aggregate visual features, the foreground visual features, and the global visual features according to the query visual features to obtain an associated example image of the query image; Inputting the query image, the associated example image and the associated example image processing result into the visual language model, and outputting the image processing result of the query image; The model training loss value is calculated based on the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result; Using the model training loss value to update the model parameters of the visual language model until the model training end condition is reached, thereby obtaining the trained visual language model; Wherein, performing step-by-step similarity screening in the order of the aggregate visual feature, the foreground visual feature, and the global visual feature according to the query visual feature to obtain an associated example image of the query image includes: Performing similarity screening on the query visual feature and each of the aggregated visual features to obtain a candidate aggregated visual feature associated with the query visual feature; Performing similarity screening on the query visual feature and the foreground visual feature corresponding to the candidate aggregated visual feature to obtain a candidate foreground visual feature associated with the query visual feature; Performing similarity screening on the query visual feature and the global visual feature corresponding to the candidate foreground visual feature to obtain an associated global visual feature associated with the query visual feature; The example image corresponding to the associated global visual feature is used as the associated example image.

2. The visual language model training method according to claim 1, characterized in that: The query visual feature and each of the aggregated visual features are screened for similarity to obtain a candidate aggregated visual feature associated with the query visual feature, including: Establishing semantic associations between each of the aggregated visual features and the query visual feature, respectively, and calculating a first probability value of the aggregated visual feature appearing in the query visual feature; The aggregated visual feature whose first likelihood value is greater than a first threshold is used as the candidate aggregated visual feature.

3. The visual language model training method according to claim 2, characterized in that: Establishing semantic associations between each of the aggregated visual features and the query visual feature respectively, and calculating a first possibility value of the aggregated visual feature appearing in the query visual feature, including: Using a cross attention mechanism to construct first semantic association information between the aggregated visual feature and the query visual feature; The first likelihood value is predicted based on the first semantic association information and the number of types of the aggregated visual features in the image data set.

4. The visual language model training method according to claim 1, characterized in that: The query visual feature and the foreground visual feature corresponding to the candidate aggregate visual feature are screened for similarity to obtain a candidate foreground visual feature associated with the query visual feature, including: According to the foreground type label corresponding to the candidate aggregated visual feature, query and obtain the corresponding first candidate foreground visual feature; Establishing semantic associations between each of the first candidate foreground visual features and the query visual feature, respectively, and calculating a second probability value of the first candidate foreground visual feature appearing in the query visual feature; The first candidate foreground visual feature whose second likelihood value is greater than a second threshold is used as the candidate foreground visual feature.

5. The visual language model training method according to claim 4, characterized in that: Establishing semantic associations between each of the first candidate foreground visual features and the query visual feature respectively, and calculating a second probability value of the first candidate foreground visual feature appearing in the query visual feature, including: Using a cross-attention mechanism to construct second semantic association information between the first candidate foreground visual feature and the query visual feature; The second likelihood value is predicted based on the second semantic association information and the number of types of the aggregated visual features in the image data set.

6. The visual language model training method according to claim 1, characterized in that: The query visual feature and the global visual feature corresponding to the candidate foreground visual feature are similarly screened to obtain an associated global visual feature associated with the query visual feature, including: Obtaining the global visual feature corresponding to the candidate foreground visual feature, recorded as the first global visual feature; Establishing semantic associations between each of the first global visual features and the query visual feature respectively, and calculating similarity values ​​between each of the first global visual features and the query visual feature; According to the descending order of the similarity values, a first first number of the first global visual features are adopted as the associated global visual features.

7. The visual language model training method according to claim 1, characterized in that: The step of calculating the similarity loss value between the associated example image and the query image comprises: A loss value between the candidate aggregated visual feature and the true label of the query image is calculated as the similarity loss value.

8. The visual language model training method according to claim 1, characterized in that: The step of calculating the similarity loss value between the associated example image and the query image comprises: Calculating the loss between the candidate aggregated visual feature and the true label of the query image to obtain a first similarity loss value; Calculating the loss between the candidate foreground visual feature and the true label of the query image to obtain a second similarity loss value; Calculating the loss between the associated global visual feature and the true label of the query image to obtain a third similarity loss value; The similarity loss value is calculated according to the first similarity loss value, the second similarity loss value and the third similarity loss value.

9. The visual language model training method according to claim 1, characterized in that: Inputting the query image, the associated example image, and the associated example image processing result into the visual language model, and outputting the image processing result of the query image, comprises: According to the similarity with the query image from large to small, taking the first first number of the associated example images and the associated example image processing results of the associated example images as positive sample pairs; According to the similarity with the query image from small to large, the first number of example images and the example image processing results of the example images are used as negative sample pairs; The query image, the positive sample pair and the negative sample pair are input into the visual language model to obtain a positive sample image processing result based on the positive sample pair for the query image and a negative sample image processing result based on the negative sample pair for the query image.

10. The visual language model training method according to claim 9, characterized in that: The calculation steps of the processing loss value of the image processing result are as follows: Calculate the contrast loss value according to the positive sample image processing result, the negative sample image processing result and the label image processing result corresponding to the query image; The contrast loss value is used as the processing loss value.

11. An image processing method, characterized in that: include: Acquire a query image and an image data set, extract global visual features and foreground visual features of example images in the image data set, and aggregate similar foreground visual features to obtain aggregated visual features; Extracting query visual features of the query image using a visual prompter in a visual language model, and performing step-by-step similarity screening in the order of the aggregate visual features, the foreground visual features, and the global visual features according to the query visual features to obtain an associated example image of the query image; Inputting the query image, the associated example image and the associated example image processing result into the visual language model, and outputting the image processing result of the query image; The model training loss value is calculated based on the similarity loss value between the associated example image and the query image and the processing loss value of the image processing result; Using the model training loss value to update the model parameters of the visual language model until the model training end condition is reached, thereby obtaining the trained visual language model; Inputting an input image into the trained visual language model, after querying the image data set to obtain a target example image associated with the input image, performing visual language calculation on the input image, the target example image, and a prompt generated by an example image processing result of the target example image, to obtain an image processing result of the input image; Wherein, performing step-by-step similarity screening in the order of the aggregate visual feature, the foreground visual feature, and the global visual feature according to the query visual feature to obtain an associated example image of the query image includes: Performing similarity screening on the query visual feature and each of the aggregated visual features to obtain a candidate aggregated visual feature associated with the query visual feature; Performing similarity screening on the query visual feature and the foreground visual feature corresponding to the candidate aggregated visual feature to obtain a candidate foreground visual feature associated with the query visual feature; Performing similarity screening on the query visual feature and the global visual feature corresponding to the candidate foreground visual feature to obtain an associated global visual feature associated with the query visual feature; The example image corresponding to the associated global visual feature is used as the associated example image.

12. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the visual language model training method according to any one of claims 1 to 10 or the image processing method according to claim 11 are implemented.

13. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visual language model training method according to any one of claims 1 to 10 or the image processing method according to claim 11 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the visual language model training method according to any one of claims 1 to 10 or the image processing method according to claim 11 are implemented.

Images