Image processing methods, apparatus and computer equipment

Abstract

This application relates to an image processing method, apparatus, and computer device. The method includes: acquiring an initial feature map of the image to be processed and determining the task features of the image processing task based on an image processing task; performing multi-scale convolution on the initial feature map based on the multi-scale structure of a large visual language model to obtain target feature maps at multiple scales; extracting region-level features from each target feature map according to the region-level structure of the large visual language model to obtain region-level feature vectors; and performing image processing on the image to be processed based on the region-level feature vectors and task features to obtain the image processing result corresponding to the task features. This method can improve the accuracy of the large visual language model in dense prediction tasks.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, and computer device. Background Technology

[0002] With the development of artificial intelligence technology, the application of visual language large models is increasing. By combining information from text and images, visual language large models enable a closer connection between natural language descriptions and images.

[0003] In traditional techniques, large visual language models extract features from input images and descriptive text using a backbone network and a text encoder to obtain global feature maps of the images and question text. The global feature maps corresponding to the images are then analyzed and processed according to the question text. The image processing result corresponding to the question text is determined based on the similarity between the global feature maps and the question text. For example, image search can be performed on multiple input images based on the target category in the question text, or the image category of the input image can be determined based on the question text.

[0004] However, in traditional techniques, the application of large visual language models suffers from poor accuracy in dense prediction tasks because the backbone network performs feature analysis on the input image at a relatively singular scale, only extracting features at the global level. Summary of the Invention

[0005] Therefore, it is necessary to provide an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0006] In a first aspect, this application provides an image processing method, comprising:

[0007] Based on the image processing task, the initial feature map of the image to be processed is obtained and the task features of the image processing task are determined.

[0008] Based on the multi-scale structure of the large visual language model, the initial feature map is convolved at multiple scales to obtain target feature maps at multiple scales.

[0009] Based on the region-level structure of the visual language big model, region-level feature extraction is performed on each of the target feature maps to obtain region-level feature vectors;

[0010] Based on the region-level feature vector and the task features, image processing is performed on the image to be processed to obtain the image processing result corresponding to the task features.

[0011] In one embodiment, the multi-scale structure includes an initial convolutional structure, a first downsampling structure, and a second downsampling structure; the target feature map includes a first-scale feature map, a second-scale feature map, and a third-scale feature map; the multi-scale structure based on a large visual language model performs multi-scale convolution on the initial feature map to obtain target feature maps of multiple scales, including:

[0012] The initial feature map is subjected to a convolutional transformation based on the initial convolutional structure to obtain the first scale feature map;

[0013] The first scale feature map is sampled and its distribution corrected according to the first downsampling structure to obtain the second scale feature map;

[0014] The second scale feature map is sampled and its distribution corrected according to the second downsampling structure to obtain the third scale feature map.

[0015] In one embodiment, if the image processing result is to determine the location of a target object, the step of extracting region-level features from each target feature map according to the region-level structure of the visual language big model to obtain a region-level feature vector includes:

[0016] The initial feature map is identified based on the region-level structure of the visual language big model, and candidate bounding boxes corresponding to the target objects in the task features are determined in the initial feature map.

[0017] Based on the region-level structure of the visual language big model and the candidate bounding boxes in the initial feature map, the bounding boxes of each target feature map are mapped, and the initial feature map after bounding box mapping is cropped according to the cropping structure to obtain the unaligned region features at each scale.

[0018] Based on the alignment structure, the unaligned region features at each scale are scale aligned to obtain a region-level feature vector with the same feature dimension as the text encoding corresponding to the task text.

[0019] The location of the target object is obtained by performing similarity matching between the region-level feature vector and the text encoding of the target object in the task text.

[0020] In one embodiment, before acquiring the initial feature map of the image to be processed and determining the task features of the image processing task based on the image processing task, the method further includes:

[0021] Obtain a set of sample images, category labels corresponding to each sample image, location labels corresponding to each sample image in the set of sample images, category information corresponding to each location label, and an initial large visual language model to be trained;

[0022] Based on the initial visual language big model, data processing is performed on each of the sample images and the category labels of each of the sample images to obtain the first feature map corresponding to the sample image and the category embedding corresponding to each of the category labels;

[0023] Based on the multi-scale structure of the initial visual language large model, the initial feature map is subjected to multi-scale convolution transformation to obtain second feature maps of multiple scales. Then, according to the region-level structure of the initial visual language large model, region-level feature extraction is performed on each second feature map to obtain target region-level feature vectors.

[0024] The multi-scale structure and region-level structure of the initial visual language large model are trained based on the region vector construction structure, the category information, the category embedding, the target region-level feature vector, and the region-level loss function construction structure to obtain the trained visual language large model.

[0025] In one embodiment, the region-level structure includes a cropping structure and an alignment structure; the step of extracting region-level features from each of the second feature maps based on the region-level structure of the initial visual language large model to obtain a target region-level feature vector includes:

[0026] Based on the region-level structure of the initial visual language large model, the second feature map is mapped with a bounding box, and the second feature map after bounding box mapping is cropped according to the cropping structure to obtain the target misaligned region features at each scale.

[0027] Based on the alignment structure, the unaligned region features of the target at each scale are scale aligned to obtain a target region-level feature vector with the same feature dimension as the category embedding.

[0028] In one embodiment, the alignment structure further includes a linear projection structure; the step of performing scale alignment on the unaligned target region features at each scale based on the alignment structure to obtain a target region-level feature vector with the same feature dimension as the category embedding includes:

[0029] Based on the alignment structure, the spatial dimension max pooling calculation is performed on the features of the unaligned regions of the target at each of the aforementioned scales to obtain an initial region-level feature vector;

[0030] Based on the linear projection structure, the initial region-level feature vectors are projected according to the dimension corresponding to the category encoding to obtain the target region-level feature vector.

[0031] In one embodiment, training the multi-scale structure and region-level structure of the initial visual language large model based on the region vector construction structure of the initial visual language large model, the category information, the category embedding, the target region-level feature vector, and the region-level loss function construction structure to obtain the trained visual language large model includes:

[0032] Based on the region vector construction structure of the initial visual language large model, the category information corresponding to each of the location annotations is encoded to obtain the category vector corresponding to each of the category information;

[0033] Based on the structure constructed by the region-level loss function, the loss value is calculated for the target region-level feature vector, category embedding and category vector at each scale to obtain the loss value at each scale.

[0034] The target loss value is determined based on the loss value at each scale, and the multi-scale structure and region-level structure of the initial visual language large model are trained based on the target loss value until the target loss value meets the preset loss condition, thus obtaining the trained visual language large model.

[0035] In one embodiment, the step of acquiring an initial feature map of the image to be processed and determining the task features of the image processing task based on the image processing task includes:

[0036] Based on the image processing task, obtain the image to be processed and the corresponding task text of the image to be processed;

[0037] Based on the backbone network of the visual language model, feature extraction is performed on the image to be processed and the task text to obtain the initial feature map corresponding to the image to be processed.

[0038] The task text is encoded using the text encoder of the visual language big data model to obtain the text encoding;

[0039] The task features corresponding to the task text are determined based on the text encoding.

[0040] Secondly, this application also provides an image processing apparatus, comprising:

[0041] The first acquisition module is used to acquire an initial feature map of the image to be processed and determine the task features of the image processing task based on the image processing task.

[0042] The first convolution module is used to perform multi-scale convolution on the initial feature map based on the multi-scale structure of the large visual language model to obtain target feature maps of multiple scales.

[0043] The first feature extraction module is used to perform regional-level feature extraction on each of the target feature maps according to the regional-level structure of the visual language big model, so as to obtain regional-level feature vectors.

[0044] The image processing module is used to perform image processing on the image to be processed based on the region-level feature vector and the task features, and obtain the image processing result corresponding to the task features.

[0045] In one embodiment, the multi-scale structure includes an initial convolutional structure, a first downsampling structure, and a second downsampling structure; the target feature map includes a first-scale feature map, a second-scale feature map, and a third-scale feature map; the first convolutional module is specifically used to perform a convolutional transformation on the initial feature map according to the initial convolutional structure to obtain the first-scale feature map;

[0046] The first scale feature map is sampled and its distribution corrected according to the first downsampling structure to obtain the second scale feature map;

[0047] The second scale feature map is sampled and its distribution corrected according to the second downsampling structure to obtain the third scale feature map.

[0048] In one embodiment, if the image processing result is to determine the location of a target object, the first feature extraction module is specifically used to identify the initial feature map according to the region-level structure of the visual language big model, and to determine the candidate bounding box corresponding to the target object in the task features in the initial feature map;

[0049] Based on the region-level structure of the visual language big model and the candidate bounding boxes in the initial feature map, the bounding boxes of each target feature map are mapped, and the initial feature map after bounding box mapping is cropped according to the cropping structure to obtain the unaligned region features at each scale.

[0050] Based on the alignment structure, the unaligned region features at each scale are scale aligned to obtain a region-level feature vector with the same feature dimension as the text encoding corresponding to the task text.

[0051] The location of the target object is obtained by performing similarity matching between the region-level feature vector and the text encoding of the target object in the task text.

[0052] In one embodiment, the device further includes:

[0053] The second acquisition module is used to acquire a sample image set, category labels corresponding to each sample image, location labels corresponding to each sample image in the sample image set, category information corresponding to each location label, and an initial visual language large model to be trained.

[0054] The embedding module is used to perform data processing on each of the sample images and the category labels of each of the sample images according to the initial visual language large model, so as to obtain the first feature map corresponding to the sample image and the category embedding corresponding to each of the category labels;

[0055] The second convolution module is used to perform multi-scale convolution transformation on the initial feature map based on the multi-scale structure of the initial visual language model to obtain second feature maps of multiple scales, and to perform region-level feature extraction on each second feature map according to the region-level structure of the initial visual language model to obtain target region-level feature vectors.

[0056] The training module is used to train the multi-scale structure and region-level structure of the initial visual language large model based on the region vector construction structure, the category information, the category embedding, the target region-level feature vector, and the region-level loss function construction structure, so as to obtain the trained visual language large model.

[0057] In one embodiment, the region-level structure includes a cropping structure and an alignment structure; the second convolution module is specifically used to map the second feature map to a bounding box according to the region-level structure of the initial visual language large model, and to crop the second feature map after the bounding box mapping according to the cropping structure to obtain the target unaligned region features at each scale;

[0058] Based on the alignment structure, the unaligned region features of the target at each scale are scale aligned to obtain a target region-level feature vector with the same feature dimension as the category embedding.

[0059] In one embodiment, the alignment structure further includes a linear projection structure; the second convolution module is specifically used to perform spatial dimension max pooling calculation on the unaligned region features of the target at each scale according to the alignment structure to obtain an initial region-level feature vector;

[0060] Based on the linear projection structure, the initial region-level feature vectors are projected according to the dimension corresponding to the category encoding to obtain the target region-level feature vector.

[0061] In one embodiment, the training module is specifically used to encode the category information corresponding to each location annotation based on the region vector construction structure of the initial visual language large model, so as to obtain the category vector corresponding to each category information;

[0062] Based on the structure constructed by the region-level loss function, the loss value is calculated for the target region-level feature vector, category embedding and category vector at each scale to obtain the loss value at each scale.

[0063] The target loss value is determined based on the loss value at each scale, and the multi-scale structure and region-level structure of the initial visual language large model are trained based on the target loss value until the target loss value meets the preset loss condition, thus obtaining the trained visual language large model.

[0064] In one embodiment, the device further includes:

[0065] The third acquisition module is used to acquire the image to be processed and the task text corresponding to the image to be processed based on the image processing task.

[0066] The second feature extraction module is used to extract features from the image to be processed and the task text based on the backbone network of the visual language large model to obtain the initial feature map corresponding to the image to be processed.

[0067] The encoding module is used to encode the task text according to the text encoder of the visual language large model to obtain the text encoding;

[0068] The determination module is used to determine the task features corresponding to the task text based on the text encoding.

[0069] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0070] Based on the image processing task, the initial feature map of the image to be processed is obtained and the task features of the image processing task are determined.

[0071] Based on the multi-scale structure of the large visual language model, the initial feature map is convolved at multiple scales to obtain target feature maps at multiple scales.

[0072] Based on the region-level structure of the visual language big model, region-level feature extraction is performed on each of the target feature maps to obtain region-level feature vectors;

[0073] Based on the region-level feature vector and the task features, image processing is performed on the image to be processed to obtain the image processing result corresponding to the task features.

[0074] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0075] Based on the image processing task, the initial feature map of the image to be processed is obtained and the task features of the image processing task are determined.

[0076] Based on the multi-scale structure of the large visual language model, the initial feature map is convolved at multiple scales to obtain target feature maps at multiple scales.

[0077] Based on the region-level structure of the visual language big model, region-level feature extraction is performed on each of the target feature maps to obtain region-level feature vectors;

[0078] Based on the region-level feature vector and the task features, image processing is performed on the image to be processed to obtain the image processing result corresponding to the task features.

[0079] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0080] Based on the image processing task, the initial feature map of the image to be processed is obtained and the task features of the image processing task are determined.

[0081] Based on the multi-scale structure of the large visual language model, the initial feature map is convolved at multiple scales to obtain target feature maps at multiple scales.

[0082] Based on the region-level structure of the visual language big model, region-level feature extraction is performed on each of the target feature maps to obtain region-level feature vectors;

[0083] Based on the region-level feature vector and the task features, image processing is performed on the image to be processed to obtain the image processing result corresponding to the task features.

[0084] The aforementioned image processing methods, apparatuses, computer devices, computer-readable storage media, and computer program products, through multi-scale convolution of the image to be processed, enable the model to capture the features of the image at different scales, thereby gaining a more comprehensive understanding of the image content. Further feature extraction of the target feature map is achieved through a region-level structure, yielding region-level feature vectors. This allows the visual language model to more accurately focus on key regions in the image to be processed, extracting local features closely related to the task text. This enhances the visual language model's ability to perceive the position of objects of different sizes, enabling it to flexibly adapt to different types of visual tasks and effectively improve the accuracy of the visual language model in dense prediction tasks. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a flowchart illustrating an image processing method in one embodiment;

[0087] Figure 2 This is a flowchart illustrating the process of determining a multi-scale feature map in one embodiment;

[0088] Figure 3 This is a schematic diagram of a multi-scale structure in one embodiment;

[0089] Figure 4 This is a schematic diagram of the downsampling structure in one embodiment;

[0090] Figure 5 This is a flowchart illustrating task processing within a region-level structure in one embodiment.

[0091] Figure 6 This is a schematic diagram illustrating the process of training an initial large visual language model in one embodiment.

[0092] Figure 7 This is a schematic diagram illustrating the embedding process of a text encoder in one embodiment.

[0093] Figure 8 This is a flowchart illustrating the data processing steps for a region-level structure in one embodiment.

[0094] Figure 9 This is a schematic diagram of a region-level structure in one embodiment;

[0095] Figure 10 This is a flowchart illustrating the data processing of the alignment structure in one embodiment.

[0096] Figure 11 This is a schematic diagram of a linear projection structure in one embodiment;

[0097] Figure 12 This is a schematic diagram illustrating the process of calculating the loss of the initial large visual language model in one embodiment;

[0098] Figure 13 A schematic diagram illustrating one-hot encoding of a region vector construction structure in one embodiment;

[0099] Figure 14 This is a schematic diagram of the construction structure of the region-level loss function in one embodiment;

[0100] Figure 15 This is a schematic diagram of the visual language large model architecture in one embodiment;

[0101] Figure 16 This is a flowchart illustrating an example of obtaining an initial feature map and task features in another embodiment;

[0102] Figure 17 This is a schematic diagram of the process of obtaining an initial feature map in one embodiment;

[0103] Figure 18 This is a structural block diagram of an image processing device in one embodiment;

[0104] Figure 19 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0105] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0106] In traditional techniques, the backbone network of large visual language models has a large receptive field for feature extraction and analysis of input images, but the scale of image feature analysis is relatively singular. Therefore, traditional visual language models can only analyze global features of the input image, resulting in poor ability to capture local details and a lack of perception of local regions. This leads to poor zero-shot generalization ability of traditional large visual language models in dense prediction tasks, such as, but not limited to, object detection and image segmentation. Therefore, to improve the location perception ability of large visual language models for local regions and enable them to recognize local information, this application proposes a novel image processing method.

[0107] In one embodiment, such as Figure 1 As shown, an image processing method is provided. This embodiment illustrates the method applied to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0108] Step 102: Based on the image processing task, obtain the initial feature map of the image to be processed and determine the task features of the image processing task.

[0109] In this embodiment, the terminal responds to the user's triggering operation for an image processing task by receiving the image to be processed input by the user. First, the terminal sends the image to be processed into a large visual language model, and the image encoder based on the large visual language model obtains the initial feature map of the image to be processed. At the same time, the terminal determines the task features according to the specific processing task. For example, if the image processing task is to identify a specific object in the image, then the task features include the object's attributes, classification, and other feature information.

[0110] Step 104: Perform multi-scale convolution on the initial feature map based on the multi-scale structure of the large visual language model to obtain target feature maps of multiple scales.

[0111] Among them, the visual language big model is an improvement on the CLIP model architecture, which adds multi-scale structure and region-level structure to obtain the visual language big model.

[0112] In this embodiment, the terminal performs multi-scale convolution operations on the initial feature map using convolution kernels of different sizes (e.g., 3x3, 5x5, etc.) within the multi-scale structure of the visual language large model. Each convolution kernel moves across the space of the initial feature map, thereby extracting local structural information of the image to be processed. Feature extraction is performed on the initial feature map according to different levels and ranges to obtain target feature maps at multiple scales. Each convolution operation generates a new target feature map, where multiple target feature maps at different scales contain image features captured at different scales. These target feature maps help the model understand different levels of image features, including information from details to the global picture, enabling the visual language large model to have good perceptual effects on both global and local information in the image to be processed.

[0113] Step 106: Based on the regional structure of the visual language big model, perform regional feature extraction on the feature maps of each target to obtain regional feature vectors.

[0114] In this embodiment, the terminal searches for regions of interest (e.g., the location of an object) on target feature maps at various scales using the region-level structure of a large visual language model. These regions of interest can be determined using a sliding window or a region proposal network. Based on the information in the task features, region-level features are extracted from each region. Specifically, region-level feature extraction is performed on target feature maps at different scales. This includes cropping the features within each region of interest and aligning these cropped features to a uniform size using ROI Pooling and ROI Align operations. This ensures that regions of different sizes can generate feature representations with uniform dimensions, thereby obtaining region-level feature vectors extracted from the target feature maps at various scales.

[0115] Step 108: Perform image processing on the image to be processed based on the region-level feature vector and task features to obtain the image processing result corresponding to the task features.

[0116] In this embodiment, the terminal uses regional feature vectors and task features to process image data to obtain the final image processing result corresponding to the task features. Specifically, the visual language big model also includes a post-processing module, such as a fully connected network, to perform tasks such as classification, segmentation, and detection. This post-processing module uses regional feature vectors to predict the output of the task, thereby obtaining an image processing result that meets the user's needs.

[0117] In the image processing method described above, by convolving the image to be processed through a multi-scale structure, the model can capture the features of the image at different scales, thereby gaining a more comprehensive understanding of the image content. Further feature extraction of the target feature map is performed through a region-level structure to obtain region-level feature vectors. This enables the visual language model to more accurately focus on key regions in the image to be processed, extract local features closely related to the task text, and improve the visual language model's ability to perceive the position of objects of different sizes. As a result, it can flexibly adapt to different types of visual tasks and effectively improve the accuracy of the visual language model in dense prediction tasks.

[0118] In an exemplary embodiment, the multi-scale structure includes an initial convolutional structure, a first downsampling structure, and a second downsampling structure, and the target feature map includes a first-scale feature map, a second-scale feature map, and a third-scale feature map; as shown Figure 2 As shown, step 104 includes steps 202 to 206. Wherein:

[0119] Step 202: Perform convolution transformation on the initial feature map according to the initial convolution structure to obtain the first scale feature map.

[0120] In the embodiments of this application, such as Figure 3 As shown, the initial convolutional structure can be a 1x1 convolutional kernel. The terminal performs convolutional transformation on the initial feature map based on the inter-channel relationship between features of the initial feature map learned by the convolutional kernel during training (i.e., the weighted combination between different channels to generate a new feature representation), extracts the high-level abstract features in the initial feature map, and then obtains the first-scale feature map (first-scale feature map) at the first scale.

[0121] Step 204: Sample and correct the distribution of the first scale feature map according to the first downsampling structure to obtain the second scale feature map.

[0122] In the embodiments of this application, such as Figure 3 As shown, the terminal inputs the first-scale feature map output from the initial convolutional structure into the first downsampling module, specifically, as follows: Figure 4As shown, the first downsampling structure includes a 2x2 convolutional kernel, a batch normalization structure, and an activation function (ReLU, a type of activation function). The first downsampling structure first uses a 2x2 convolutional kernel to downsample the first-scale feature map, then performs distribution correction on the downsampled first-scale feature map according to the batch normalization structure, and finally uses the activation function ReLU to improve the expressive power of the first-scale feature map to obtain the second-scale feature map.

[0123] Step 206: Sample and correct the distribution of the second-scale feature map according to the second downsampling structure to obtain the third-scale feature map.

[0124] In this embodiment, the terminal samples and corrects the distribution of the second-scale feature map according to the same principle as step 204, based on the second downsampling structure, to obtain the third-scale feature map. The detailed process of sampling and distribution correction will not be described in this embodiment.

[0125] In this embodiment, the initial feature map is compressed and abstracted through the convolution transformation of the initial convolution structure. Then, sampling and distribution correction are performed according to the first and second downsampling structures, which can extract more high-level feature information. This allows the visual language big model to extract target feature maps of different scales from the image to be processed, obtain feature representations of different abstract levels, improve the comprehensiveness of feature extraction of the image to be processed by the visual language big model, and improve the accuracy of subsequent image processing.

[0126] In one exemplary embodiment, such as Figure 5 As shown, if the image processing result is to determine the location of the target object, step 106 includes steps 502 to 508. Wherein:

[0127] Step 502: Identify the initial feature map based on the regional structure of the visual language big model, and determine the candidate bounding box corresponding to the target object in the task features in each target feature map.

[0128] In this embodiment of the application, firstly, the terminal extracts feature representations related to the target object from the initial feature map based on the visual language big model. The terminal uses the region-level structure of the visual language big model to identify the target object and generate bounding boxes on the initial feature map. It performs preliminary identification over a large area in the initial feature map to obtain multiple candidate bounding boxes related to the target object in the task features.

[0129] Step 504: Map the bounding boxes of each target feature map to the region-level structure of the visual language big model and the candidate bounding boxes in the initial feature map, and crop the initial feature map after the bounding box mapping according to the cropping structure to obtain the unaligned region features at each scale.

[0130] In this embodiment, the terminal maps each candidate bounding box to the same scale as each target feature map, thus obtaining multiple bounding box mapped target feature maps. Then, according to the clipping structure in the region-level structure, the terminal clips out the high-quality feature map corresponding to each bounding box from the target feature map after bounding box mapping. Since different bounding boxes may correspond to different scales, the clipped target feature maps retain the scale differences. Therefore, after the terminal performs bounding box mapping on the target feature map, it obtains the unaligned region features at each scale.

[0131] Step 506: Based on the alignment structure, scale-align the unaligned region features at each scale to obtain a region-level feature vector with the same feature dimension as the text encoding corresponding to the task text.

[0132] In this embodiment, the terminal performs scale alignment on the cropped unaligned region features using an alignment structure. Based on the text encoding of the target object in the task text, the feature map pixels of the cropped unaligned region features are mapped to a fixed spatial size, resulting in a region-level feature vector with the same feature dimension as the text encoding of the target object.

[0133] Step 508: Perform similarity matching between the region-level feature vector and the text encoding of the target object in the task text to obtain the location of the target object.

[0134] In this embodiment, the terminal performs similarity matching between multiple scale-aligned region-level feature vectors and the embedded encoding of the target object in the task text by calculating the cosine similarity or other similarity measures between vectors. The terminal identifies the candidate bounding box with the highest matching score as the location of the target object, and this location information can be used for further image processing tasks, such as object detection and object recognition.

[0135] In this embodiment, candidate bounding boxes are generated through a region-level structure for final similarity matching. By combining visual and linguistic features through the region-level structure, the visual language big model can understand and process the image to be processed at both global and local semantic levels, achieving accurate local target object location determination and improving the accuracy of the visual language big model in dense prediction tasks.

[0136] In one exemplary embodiment, such as Figure 6 As shown, before step 102, a large visual language model needs to be trained. This method also includes steps 602 to 608. Among them:

[0137] Step 602: Obtain the sample image set, the category label corresponding to each sample image, the location label corresponding to each sample image in the sample image set, the category information corresponding to each location label, and the initial visual language large model to be trained.

[0138] In this embodiment, the sample image set can be obtained by mixing an object detection dataset and an image segmentation dataset. Specifically, the object detection dataset can be a COCO detection dataset or a dataset collected for a specific business scenario; the object detection dataset can be represented as... ,in This represents the i-th image in the object detection dataset. Represents the coordinates of multiple bounding boxes corresponding to the i-th image. That is, the location label corresponding to the sample image. Represents the category information of the i-th image. The category names for each object detection dataset are denoted as... That is, the category label corresponding to each sample image.

[0139] The image segmentation dataset can be the ade20k dataset or a dataset collected from a business scenario; the image segmentation dataset can be represented as... ,in This represents the i-th image in the dataset. Let the multiple masks corresponding to the i-th image be represented as follows: The masks are transformed into outer matrices, denoted as... That is, the location label corresponding to the sample image. Represents the category information of the i-th image. The category names for each image segmentation dataset are denoted as... That is, the category label corresponding to each sample image.

[0140] After mixing the object detection dataset and the image segmentation dataset, a hybrid dataset is obtained, which is the sample image set. The hybrid dataset is represented as follows: ,in This represents the total number of images in the object detection dataset. This represents the total number of images in the image segmentation dataset; the category names of the mixed dataset are denoted as... Let the total amount be denoted as n.

[0141] Step 604: Based on the initial visual language big model, perform data processing on each sample image and the category label of each sample image to obtain the first feature map corresponding to the sample image and the category embedding corresponding to each category label.

[0142] In this embodiment, firstly, the terminal freezes the pre-trained weights of the initial visual language large-scale model. These pre-trained weights are the weights of each part in the original structure of the visual language large-scale model. Then, fine-tuning training is performed on the multi-scale structure and region-level structure of the initial visual language large-scale model. The terminal inputs each sample image from the sample image set into the backbone network of the visual language large-scale model, and obtains the first feature map corresponding to each sample image through the backbone network, denoted as... For category labeling, such as Figure 7 As shown, the terminal inputs the category labels (i.e., category names) of each sample image into the text encoder, and obtains the category embeddings corresponding to the category labels through the text encoder, denoted as . .

[0143] Step 606: Perform multi-scale convolution transformation on the initial feature map based on the multi-scale structure of the initial visual language large model to obtain second feature maps of multiple scales, and extract regional features from each second feature map according to the regional structure of the initial visual language large model to obtain the target regional feature vector.

[0144] In this embodiment, firstly, the terminal performs convolutional transformation on the first feature map using the multi-scale structure in the initial visual language large model to generate second feature maps at multiple scales. These first feature maps capture different visual details and semantic information of the sample image at different scales. Then, based on the region-level structure of the initial visual language large model, the terminal performs region-level feature extraction on the second feature map at each scale, extracting the target region-level feature map from the pyramid feature map. For example, through object detection or region proposal networks, the target region-level feature vector is extracted from the second feature map, resulting in a target region-level feature vector that embodies the semantic and positional information of each bounding box in local semantics.

[0145] Step 608: Based on the region vector construction structure, category information, category embedding, target region-level feature vector, and region-level loss function of the initial visual language large model, train the multi-scale structure and region-level structure of the initial visual language large model to obtain the trained visual language large model.

[0146] In this embodiment, the terminal constructs the loss function and calculates the loss value for category information, category embedding, and target region-level feature vectors based on the region vector construction structure and the region-level loss function construction structure. The terminal then fine-tunes the model parameters of the initial visual language model using the Stochastic Gradient Descent (SGD) algorithm or the Adaptive Moment Estimation (Adam) algorithm to obtain the trained visual language model.

[0147] In this embodiment, through multi-scale convolutional transformation and region-level feature extraction, detailed information of the image can be captured at different scales, and this information is aligned with the text embedding, so that the visual language large model can effectively recognize different scales and positions, and improve the ability to perceive the position of objects of different sizes.

[0148] In one exemplary embodiment, the region-level structure includes a clipping structure and an alignment structure, the alignment structure further including a linear projection structure; such as Figure 8 As shown, step 606 includes steps 802 to 804. Wherein:

[0149] Step 802: Map the bounding boxes of each second feature map according to the region-level structure of the initial visual language large model, and crop the second feature map after the bounding box mapping according to the cropping structure to obtain the target misaligned region features at each scale.

[0150] In this embodiment of the application, the terminal follows the same principle as in step 504, such as... Figure 9 As shown, through the region-level structure of the initial visual language large model, the second feature maps at different scales are mapped and cropped according to the position annotations corresponding to the sample images to obtain the target misaligned region features. The specific process will not be elaborated in this embodiment. Taking three scales as an example, the bounding box coordinates (position annotations) corresponding to the sample images are mapped to the corresponding positions of the feature maps at the first, second, and third scales, that is, ,in , , I represents the total number of regions of interest (ROIs) in each image. For object detection datasets, I=B; for image segmentation datasets, I=M. The terminal crops the second feature maps at three scales after mapping the bounding boxes according to the cropping structure, obtaining the unaligned region features corresponding to each bounding box at each scale, denoted as […]. .

[0151] Step 804: Based on the alignment structure, scale-align the unaligned region features of the target at each scale to obtain a target region-level feature vector with the same feature dimension as the category embedding.

[0152] In this embodiment, the terminal follows the same principle as in step 506, such as... Figure 9 As shown, the unaligned features of the target at each scale are scale-aligned using the alignment structure in the region-level structure to obtain a target region-level feature vector with the same feature dimension as the category embedding. The specific process will not be described in detail in this embodiment.

[0153] In this embodiment, the mapping and cropping operations of the calibration box ensure that the features of the second feature map location annotation region not only contain the semantic information of the target object, but also retain its spatial position and scale information in the sample image. The scale alignment operation enables the visual language large model to process input images of different scales and resolutions, while maintaining the consistency and comparability of features, improving the generalization ability of the visual language large model in practical applications, and improving the accuracy of image processing.

[0154] In one exemplary embodiment, the alignment structure further includes a linear projection structure; such as Figure 10 As shown, step 804 includes steps 1002 to 1004. Wherein:

[0155] Step 1002: Perform spatial dimension max pooling calculation on the features of the unaligned target regions at each scale according to the alignment structure to obtain the initial region-level feature vector.

[0156] In this embodiment, the terminal performs spatial dimension max pooling calculation on the features of the unaligned target region at each scale based on the alignment structure of the initial visual language large model, and obtains a one-dimensional feature vector, that is, the initial region-level feature vector corresponding to each feature of the unaligned target region.

[0157] Step 1004: Based on the linear projection structure, project each initial region-level feature vector according to the dimension corresponding to the category encoding to obtain the target region-level feature vector.

[0158] In the embodiments of this application, such as Figure 11 As shown, the terminal inputs the one-dimensional feature vector (initial region-level feature vector) corresponding to the features of each unaligned target region into the linear projection structure. Based on the linear projection structure, the one-dimensional feature vector is linearly projected according to the feature dimension of the category embedding, ensuring that the initial region-level feature vector of each target region can match the input of the category embedding or other specific dimensions required by subsequent tasks, thus obtaining the target one-dimensional feature vector with the same feature dimension as the category embedding, that is, the target region-level feature vector.

[0159] In this embodiment, the alignment structure and linear projection structure of the visual language big model are used to align the features and category embeddings of the unaligned target region, enabling the visual language big model to effectively process the target region information extracted from the image and transform it into a feature representation suitable for processing by each part of the visual language big model. This improves the performance and generalization ability of the visual language big model in complex visual and language tasks.

[0160] In one exemplary embodiment, such as Figure 12 As shown, step 608 includes steps 1202 to 1206. Wherein:

[0161] Step 1202: Based on the region vector of the initial visual language model, the category information corresponding to each location annotation is encoded to obtain the category vector corresponding to each category information.

[0162] In the embodiments of this application, such as Figure 13 As shown, the terminal inputs the category information corresponding to each location annotation in each sample image into the region vector construction structure of the initial visual language large model. Each location annotation contains multiple coordinates, and each coordinate has its corresponding category index, meaning each location annotation has corresponding category information. For each location annotation containing the coordinate index, the terminal performs one-hot encoding to obtain the category vector corresponding to each category information.

[0163] Step 1204: Based on the regional loss function, construct a structure to calculate the loss value of the target regional feature vector, class embedding and class vector at each scale, and obtain the loss value at each scale.

[0164] In this embodiment, the terminal inputs the target region-level feature vectors at each scale, the corresponding category embeddings, and the category vectors into the region-level loss function construction structure, and obtains the loss value for each scale through the region-level loss function construction structure. Specifically, as shown... Figure 14 As shown, for each target region-level feature vector at each scale The similarity value is calculated with each category embedding, taking cosine similarity as an example. The similarity result is denoted as... Where i=1, ..., 3, j=1, ..., I, k=1, ..., n, and k is the category index (category information); then the terminal uses a softmax operation to transform the aforementioned similarity results into a probability distribution along the dimension of the category information k, denoted as... Ultimately, the terminal is based on a probability distribution. and category vectors Using cross-entropy loss, we obtain the region-level loss function, denoted as... Where i=1,...,3, j=1,...,I, then the terminal calculates the loss value at each scale. Summing along dimension j yields the loss values ​​at each scale, denoted as . .

[0165] Step 1206: Determine the target loss value based on the loss values ​​at each scale, and train the multi-scale structure and region-level structure of the initial visual language large model based on the target loss value until the target loss value meets the preset loss condition, thus obtaining the trained visual language large model.

[0166] In this embodiment, the terminal sums the loss values ​​at each scale to obtain the final target loss value. Then, based on a stochastic gradient descent algorithm or an adaptive learning rate optimization algorithm, it fine-tunes the multi-scale and region-level structures of the initial large-scale visual language model until the target loss value meets a preset loss condition, resulting in a completed large-scale visual language model. The preset loss condition can be that the target loss value is lower than a preset loss threshold, or that the number of iterations meets a preset iteration round requirement.

[0167] In this embodiment, by integrating loss values ​​at different scales, the initial visual language large model can learn more generalized feature representations. The training of the initial visual language large model is guided by constructing probability distributions of feature similarity and category prediction through target region-level feature vectors, category embeddings, and category vectors. This allows for precise evaluation of classification accuracy and category prediction ability at each scale during training, thereby optimizing the feature learning and generalization ability of the initial visual language large model and improving its accuracy in dense prediction tasks.

[0168] In one specific embodiment, an example of a visual language large model training method is provided, such as... Figure 15 As shown, the method includes:

[0169] Step 1501: Obtain the first feature map and location annotation (gt box) of each sample image in the sample image set.

[0170] Step 1502: Process the first feature map according to the multi-scale structure to obtain the second feature map, and process the second feature map and sample labels according to the region-level structure to obtain the target region-level feature vector.

[0171] Step 1503: Encode the category labels of each sample image according to the text encoder to obtain the category embedding;

[0172] Step 1504: Based on the region vector, construct a structure to perform one-hot encoding on the category information corresponding to the location label to obtain the category vector;

[0173] Step 1505: Based on the region-level loss function, the target region-level feature vector, category embedding, and category vector are processed to obtain the target loss value of the initial visual language large model. The multi-scale structure and region-level structure of the initial visual language large model are then trained based on the target loss value.

[0174] In one exemplary embodiment, such as Figure 16 As shown, step 102 includes steps 1602 to 1606. Wherein:

[0175] Step 1602: Based on the image processing task, obtain the image to be processed and the corresponding task text.

[0176] In this embodiment, the terminal obtains the image to be processed and the corresponding task text from the actual application scenario. The image is a visual input processed by the user-input visual language big model, and the task text is the specific task or requirement to be achieved for image processing.

[0177] Step 1604: Based on the backbone network of the visual language model, feature extraction is performed on the image to be processed and the task text to obtain the initial feature map corresponding to the image to be processed.

[0178] In the embodiments of this application, such as Figure 17 As shown, the terminal uses the backbone network of the visual language big model to extract features from the image to be processed, and obtains an initial feature map of the high-level abstract representation of the image to be processed.

[0179] Step 1606: Encode the task text using the text encoder of the visual language big data model to obtain the text encoding.

[0180] In this embodiment, the terminal uses a text encoder of a visual language big model to encode the task text. The text encoder can convert the natural language task description into a vectorized representation that the visual language big model can understand and process.

[0181] Step 1608: Determine the task features corresponding to the task text based on text encoding.

[0182] In this embodiment, the terminal extracts task-related features from the text encoding based on the text mapping relationship learned by the pre-training of the visual language big model, performs semantic understanding on the task text, determines the user's intent, and obtains the task features corresponding to the task text input by the current user. These task features can be the type of image processing, such as object detection, image segmentation, phrase localization, or region-level image quality evaluation.

[0183] In this embodiment, by converting the image to be processed and the task text into corresponding feature representations, task features and initial feature maps are obtained, which provides a foundation and prerequisite for the visual language big model to effectively process visual and linguistic information in subsequent tasks.

[0184] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0185] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0186] In one exemplary embodiment, such as Figure 18 As shown, an image processing device 1800 is provided, including: a first acquisition module 1801, a first convolution module 1802, a first feature extraction module 1803, and an image processing module 1804, wherein:

[0187] The first acquisition module 1801 is used to acquire the initial feature map of the image to be processed and determine the task features of the image processing task based on the image processing task.

[0188] The first convolutional module 1802 is used to perform multi-scale convolution on the initial feature map based on the multi-scale structure of the large visual language model to obtain target feature maps of multiple scales.

[0189] The first feature extraction module 1803 is used to perform regional feature extraction on each target feature map according to the regional structure of the large visual language model, and obtain regional feature vectors.

[0190] The image processing module 1804 is used to perform image processing on the image to be processed based on the region-level feature vector and task features, and obtain the image processing result corresponding to the task features.

[0191] In one embodiment, the multi-scale structure includes an initial convolutional structure, a first downsampling structure, and a second downsampling structure, and the target feature map includes a first-scale feature map, a second-scale feature map, and a third-scale feature map; the first convolutional module 1802 is specifically used to perform a convolutional transformation on the initial feature map according to the initial convolutional structure to obtain the first-scale feature map.

[0192] The first scale feature map is sampled and its distribution corrected according to the first downsampling structure to obtain the second scale feature map;

[0193] The second-scale feature map is sampled and its distribution corrected based on the second downsampling structure to obtain the third-scale feature map.

[0194] In one embodiment, if the image processing result is to determine the location of the target object, the first feature extraction module 1803 is specifically used to identify the initial feature map according to the regional structure of the visual language big model, and determine the candidate bounding box corresponding to the target object in the task features in the initial feature map;

[0195] Based on the region-level structure of the visual language big model and the candidate bounding boxes in the initial feature map, the bounding boxes of each target feature map are mapped, and the initial feature map after the bounding boxes are mapped is cropped according to the cropping structure to obtain the unaligned region features at each scale.

[0196] Based on the alignment structure, the unaligned region features at each scale are scale aligned to obtain a region-level feature vector with the same feature dimension as the text encoding corresponding to the task text.

[0197] The location of the target object is obtained by performing similarity matching between the region-level feature vector and the text encoding of the target object in the task text.

[0198] In one embodiment, the device 1800 further includes:

[0199] The second acquisition module is used to acquire the sample image set, the category label corresponding to each sample image, the position label corresponding to each sample image in the sample image set, the category information corresponding to each position label, and the initial visual language large model to be trained.

[0200] The embedding module is used to process the data of each sample image and the category label of each sample image according to the initial visual language large model, so as to obtain the first feature map corresponding to the sample image and the category embedding corresponding to each category label;

[0201] The second convolution module is used to perform multi-scale convolution transformation on the initial feature map based on the multi-scale structure of the initial visual language model to obtain second feature maps of multiple scales, and to extract regional features from each second feature map according to the regional structure of the initial visual language model to obtain the target region-level feature vector.

[0202] The training module is used to train the multi-scale structure and region-level structure of the initial visual language large model based on the region vectors of the initial visual language large model, the structure construction of category information, category embedding, target region-level feature vectors and region-level loss function, so as to obtain the trained visual language large model.

[0203] In one embodiment, the region-level structure includes a cropping structure and an alignment structure; the second convolutional module is specifically used to map the calibration boxes of each second feature map according to the region-level structure of the initial visual language large model, and to crop the second feature map after the calibration box mapping according to the cropping structure to obtain the target unaligned region features at each scale.

[0204] Based on the alignment structure, the unaligned features of the target at each scale are scale aligned to obtain a target region-level feature vector with the same feature dimension as the category embedding.

[0205] In one embodiment, the alignment structure further includes a linear projection structure; the second convolution module is specifically used to perform spatial dimension max pooling calculation on the features of the unaligned target region at each scale according to the alignment structure to obtain an initial region-level feature vector.

[0206] Based on the linear projection structure, the initial region-level feature vectors are projected according to the dimension corresponding to the category encoding to obtain the target region-level feature vectors.

[0207] In one embodiment, the training module is specifically used to encode the category information corresponding to each location annotation based on the region vector of the initial visual language large model to obtain the category vector corresponding to each category information.

[0208] Based on the structure constructed by the regional loss function, the loss values ​​of the target regional feature vectors, class embeddings and class vectors at each scale are calculated to obtain the loss values ​​at each scale.

[0209] The target loss value is determined based on the loss value at each scale, and the multi-scale structure and region-level structure of the initial visual language large model are trained based on the target loss value until the target loss value meets the preset loss condition, thus obtaining the trained visual language large model.

[0210] In one embodiment, the device 1800 further includes:

[0211] The third acquisition module is used to acquire the image to be processed and the corresponding task text based on the image processing task.

[0212] The second feature extraction module is used to extract features from the image to be processed and the task text based on the backbone network of the visual language model, so as to obtain the initial feature map corresponding to the image to be processed.

[0213] The encoding module is used to encode the task text based on the text encoder of the visual language large model to obtain the text encoding;

[0214] The determination module is used to determine the task features corresponding to the task text based on the text encoding.

[0215] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0216] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 19 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores images to be processed. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method.

[0217] Those skilled in the art will understand that Figure 19 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0218] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0219] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0220] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0221] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0222] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0223] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0224] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Based on the image processing task, the initial feature map of the image to be processed is obtained and the task features of the image processing task are determined. Based on the multi-scale structure of the large visual language model, the initial feature map is convolved at multiple scales to obtain target feature maps at multiple scales. Based on the region-level structure of the visual language big model, region-level feature extraction is performed on each of the target feature maps to obtain region-level feature vectors; If the image processing result determines the location of the target object, the step of extracting region-level features from each target feature map according to the region-level structure of the visual language large model to obtain a region-level feature vector includes: The initial feature map is identified based on the region-level structure of the visual language big model, and candidate bounding boxes corresponding to the target objects in the task features are determined in the initial feature map. Based on the region-level structure of the visual language big model and the candidate bounding boxes in the initial feature map, the bounding boxes of each target feature map are mapped, and the initial feature map after bounding box mapping is cropped according to the cropping structure to obtain the unaligned region features at each scale. Based on the alignment structure, the unaligned region features at each scale are scale aligned to obtain a region-level feature vector with the same feature dimension as the text encoding corresponding to the task text. Based on the region-level feature vector and the task features, image processing is performed on the image to be processed to obtain the image processing result corresponding to the task features.

2. The method according to claim 1, characterized in that, The multi-scale structure includes an initial convolutional structure, a first downsampling structure, and a second downsampling structure. The target feature map includes a first-scale feature map, a second-scale feature map, and a third-scale feature map. The multi-scale structure based on the large visual language model performs multi-scale convolution on the initial feature map to obtain target feature maps of multiple scales, including: The initial feature map is subjected to a convolutional transformation based on the initial convolutional structure to obtain the first scale feature map; The first scale feature map is sampled and its distribution corrected according to the first downsampling structure to obtain the second scale feature map; The second scale feature map is sampled and its distribution corrected according to the second downsampling structure to obtain the third scale feature map.

3. The method according to claim 1, characterized in that, If the image processing result determines the location of the target object, the method further includes: The location of the target object is obtained by performing similarity matching between the region-level feature vector and the text encoding of the target object in the task text.

4. The method according to claim 1, characterized in that, Before obtaining the initial feature map of the image to be processed and determining the task features of the image processing task based on the image processing task, the method further includes: Obtain a set of sample images, category labels corresponding to each sample image, location labels corresponding to each sample image in the set of sample images, category information corresponding to each location label, and an initial large visual language model to be trained; Based on the initial visual language big model, data processing is performed on each of the sample images and the category labels of each of the sample images to obtain the first feature map corresponding to the sample image and the category embedding corresponding to each of the category labels; Based on the multi-scale structure of the initial visual language large model, the initial feature map is subjected to multi-scale convolution transformation to obtain second feature maps of multiple scales. Then, according to the region-level structure of the initial visual language large model, region-level feature extraction is performed on each second feature map to obtain target region-level feature vectors. The multi-scale structure and region-level structure of the initial visual language large model are trained based on the region vector construction structure, the category information, the category embedding, the target region-level feature vector, and the region-level loss function construction structure to obtain the trained visual language large model.

5. The method according to claim 4, characterized in that, The region-level structure includes a cropping structure and an alignment structure; the process of extracting region-level features from each of the second feature maps based on the region-level structure of the initial visual language model to obtain the target region-level feature vector includes: Based on the region-level structure of the initial visual language large model, the second feature map is mapped with a bounding box, and the second feature map after bounding box mapping is cropped according to the cropping structure to obtain the target misaligned region features at each scale. Based on the alignment structure, the unaligned region features of the target at each scale are scale aligned to obtain a target region-level feature vector with the same feature dimension as the category embedding.

6. The method according to claim 5, characterized in that, The alignment structure further includes a linear projection structure; the step of scaling the unaligned target region features at each scale based on the alignment structure to obtain a target region-level feature vector with the same feature dimension as the category embedding includes: Based on the alignment structure, the spatial dimension max pooling calculation is performed on the features of the unaligned regions of the target at each of the aforementioned scales to obtain an initial region-level feature vector; Based on the linear projection structure, the initial region-level feature vectors are projected according to the dimension corresponding to the category encoding to obtain the target region-level feature vector.

7. The method according to claim 4, characterized in that, The process of training the multi-scale and region-level structures of the initial visual language large-scale model based on the region vector construction structure of the initial visual language large-scale model, the category information, the category embedding, the target region-level feature vector, and the region-level loss function construction structure, to obtain the trained visual language large-scale model, includes: Based on the region vector construction structure of the initial visual language large model, the category information corresponding to each of the location annotations is encoded to obtain the category vector corresponding to each of the category information; Based on the structure constructed by the region-level loss function, the loss value is calculated for the target region-level feature vector, category embedding and category vector at each scale to obtain the loss value at each scale. The target loss value is determined based on the loss value at each scale, and the multi-scale structure and region-level structure of the initial visual language large model are trained based on the target loss value until the target loss value meets the preset loss condition, thus obtaining the trained visual language large model.

8. The method according to claim 1, characterized in that, The process of obtaining an initial feature map of the image to be processed and determining the task features of the image processing task based on the image processing task includes: Based on the image processing task, obtain the image to be processed and the corresponding task text of the image to be processed; Based on the backbone network of the visual language model, feature extraction is performed on the image to be processed and the task text to obtain the initial feature map corresponding to the image to be processed. The task text is encoded using the text encoder of the visual language big data model to obtain the text encoding; The task features corresponding to the task text are determined based on the text encoding.

9. An image processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire an initial feature map of the image to be processed and determine the task features of the image processing task based on the image processing task. The first convolution module is used to perform multi-scale convolution on the initial feature map based on the multi-scale structure of the large visual language model to obtain target feature maps of multiple scales. The first feature extraction module is used to perform regional-level feature extraction on each of the target feature maps according to the regional-level structure of the visual language big model, so as to obtain regional-level feature vectors. If the image processing result is to determine the location of the target object, the first feature extraction module is specifically used to identify the initial feature map according to the region-level structure of the visual language big model, and to determine the candidate bounding box corresponding to the target object in the task features in the initial feature map; Based on the region-level structure of the visual language big model and the candidate bounding boxes in the initial feature map, the bounding boxes of each target feature map are mapped, and the initial feature map after bounding box mapping is cropped according to the cropping structure to obtain the unaligned region features at each scale. Based on the alignment structure, the unaligned region features at each scale are scale aligned to obtain a region-level feature vector with the same feature dimension as the text encoding corresponding to the task text. The image processing module is used to perform image processing on the image to be processed based on the region-level feature vector and the task features, and obtain the image processing result corresponding to the task features.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images