Image processing method and device and computer equipment

Abstract

The invention relates to an image processing method and device and computer equipment. The method comprises the following steps: based on an image processing task, obtaining an initial feature map of a to-be-processed image and determining task features of the image processing task; performing multi-scale convolution on the initial feature map based on a multi-scale structure of a visual language large model to obtain a multi-scale target feature map; performing region-level feature extraction on each target feature map according to a region-level structure of the visual language large model to obtain a region-level feature vector; and performing image processing on the to-be-processed image based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature. By adopting the method, the accuracy of the visual language large model in the dense prediction task can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus and computer equipment. Background Art

[0002] With the development of artificial intelligence technology, the application of visual language big models is increasing. By combining the information of text and images, the visual language big models establish a closer connection between natural language descriptions and images.

[0003] In traditional technology, the visual language large model extracts features from the input image and description text through the backbone network and text encoder to obtain the global feature map of the image and the question text, analyzes and processes the global feature map corresponding to the image according to the question text, and determines the image processing result corresponding to the question text based on the similarity between the global feature map and the question text. For example, image search is performed on multiple input images based on the target category in the question text, or the image category of the input image is determined based on the question text.

[0004] However, in traditional technologies, for the application of large visual language models, since the backbone network performs feature analysis on the input image at a relatively single scale and can only extract image features at the global level, the large visual language model has poor accuracy in data processing for dense prediction tasks. Summary of the invention

[0005] Based on this, it is necessary to provide an image processing method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical issues.

[0006] In a first aspect, the present application provides an image processing method, comprising:

[0007] Based on the image processing task, obtaining an initial feature map of the image to be processed and determining the task features of the image processing task;

[0008] Performing multi-scale convolution on the initial feature map based on the multi-scale structure of the visual language large model to obtain target feature maps of multiple scales;

[0009] Performing regional-level feature extraction on each of the target feature maps according to the regional-level structure of the visual language large model to obtain a regional-level feature vector;

[0010] The image to be processed is subjected to image processing based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature.

[0011] In one embodiment, the multi-scale structure includes an initial convolution structure, a first downsampling structure and a second downsampling structure, and the target feature map includes a first scale feature map, a second scale feature map and a third scale feature map; the multi-scale structure based on the visual language large model performs multi-scale convolution on the initial feature map to obtain target feature maps of multiple scales, including:

[0012] Performing a convolution transformation on the initial feature map according to the initial convolution structure to obtain the first scale feature map;

[0013] Sampling and distribution correcting the first scale feature map according to the first down-sampling structure to obtain the second scale feature map;

[0014] The second scale feature map is sampled and distribution corrected according to the second down-sampling structure to obtain the third scale feature map.

[0015] In one embodiment, if the image processing result is to determine the position of the target object; performing regional-level feature extraction on each of the target feature maps according to the regional-level structure of the visual language large model to obtain a regional-level feature vector includes:

[0016] Identify the initial feature map according to the region-level structure of the visual language large model, and determine a candidate calibration box corresponding to the target object in the task feature in the initial feature map;

[0017] According to the region-level structure of the visual language large model and the candidate calibration boxes in the initial feature map, each of the target feature maps is calibrated and mapped, and the initial feature map after the calibration box mapping is cropped according to the cropping structure to obtain unaligned regional features at each scale;

[0018] Based on the alignment structure, the unaligned regional features at each scale are scale-aligned to obtain a regional-level feature vector with the same feature dimension as the text encoding corresponding to the task text;

[0019] The region-level feature vector is matched with the text encoding of the target object in the task text to obtain the position of the target object.

[0020] In one embodiment, before obtaining an initial feature map of the image to be processed and determining the task features of the image processing task based on the image processing task, the method further includes:

[0021] Obtaining a sample image set, a category label corresponding to each of the sample images, a position label corresponding to each of the sample images in the sample image set, category information corresponding to each of the position labels, and an initial visual language large model to be trained;

[0022] Performing data processing on each of the sample images and the category annotations of each of the sample images according to the initial visual language large model to obtain a first feature map corresponding to the sample image and a category embedding corresponding to each of the category annotations;

[0023] Based on the multi-scale structure of the initial visual language large model, the initial feature map is subjected to a multi-scale convolution transformation to obtain second feature maps of multiple scales, and according to the regional level structure of the initial visual language large model, regional level features are extracted from each of the second feature maps to obtain a target regional level feature vector;

[0024] The multi-scale structure and region-level structure of the initial visual language large model are trained according to the region vector construction structure of the initial visual language large model, the category information, the category embedding, the target region-level feature vector and the region-level loss function construction structure to obtain a trained visual language large model.

[0025] In one embodiment, the region-level structure includes a cropping structure and an alignment structure; and the step of extracting region-level features from each of the second feature maps according to the region-level structure of the initial visual language large model to obtain a target region-level feature vector includes:

[0026] Performing calibration frame mapping on each of the second feature maps according to the region-level structure of the initial visual language large model, and cropping the second feature maps after the calibration frame mapping according to the cropping structure to obtain target misaligned region features at each scale;

[0027] The target unaligned region features at each scale are scale-aligned based on the alignment structure to obtain a target region-level feature vector with the same feature dimension as the category embedding.

[0028] In one embodiment, the alignment structure further includes a linear projection structure; and the scale alignment of the target unaligned region features at each scale based on the alignment structure to obtain a target region-level feature vector with the same feature dimension as the category embedding includes:

[0029] Performing spatial dimension maximum pooling calculation on the target unaligned region features at each scale according to the alignment structure to obtain an initial region-level feature vector;

[0030] Based on the linear projection structure, each of the initial region-level feature vectors is projected according to the dimension corresponding to the category code to obtain a target region-level feature vector.

[0031] In one embodiment, the multi-scale structure and region-level structure of the initial visual language large model are trained according to the region vector construction structure of the initial visual language large model, the category information, the category embedding, the target region-level feature vector and the region-level loss function construction structure to obtain the trained visual language large model, including:

[0032] Encoding the category information corresponding to each of the position annotations according to the regional vector construction structure of the initial visual language large model to obtain a category vector corresponding to each of the category information;

[0033] Based on the region-level loss function, a structure is constructed to calculate the loss value of the target region-level feature vector, the category embedding and the category vector at each scale to obtain the loss value at each scale;

[0034] A target loss value is determined according to the loss values ​​at each scale, and the multi-scale structure and region-level structure of the initial visual language large model are trained based on the target loss value until the target loss value meets the preset loss condition, thereby obtaining a trained visual language large model.

[0035] In one embodiment, the step of obtaining an initial feature map of an image to be processed and determining a task feature of the image processing task based on an image processing task includes:

[0036] Based on the image processing task, obtaining an image to be processed and a task text corresponding to the image to be processed;

[0037] Extract features of the image to be processed and the task text according to the backbone network of the visual language large model to obtain an initial feature map corresponding to the image to be processed;

[0038] Encoding the task text according to the text encoder of the visual language large model to obtain a text encoding;

[0039] A task feature corresponding to the task text is determined based on the text encoding.

[0040] In a second aspect, the present application further provides an image processing device, comprising:

[0041] A first acquisition module, used to acquire an initial feature map of an image to be processed and determine a task feature of the image processing task based on an image processing task;

[0042] A first convolution module, configured to perform multi-scale convolution on the initial feature map based on the multi-scale structure of the visual language large model to obtain target feature maps of multiple scales;

[0043] A first feature extraction module is used to extract regional features from each of the target feature maps according to the regional structure of the visual language model to obtain a regional feature vector;

[0044] The image processing module is used to perform image processing on the image to be processed based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature.

[0045] In one embodiment, the multi-scale structure includes an initial convolution structure, a first down-sampling structure and a second down-sampling structure, and the target feature map includes a first scale feature map, a second scale feature map and a third scale feature map; the first convolution module is specifically used to perform a convolution transformation on the initial feature map according to the initial convolution structure to obtain the first scale feature map;

[0046] Sampling and distribution correcting the first scale feature map according to the first down-sampling structure to obtain the second scale feature map;

[0047] The second scale feature map is sampled and distribution corrected according to the second down-sampling structure to obtain the third scale feature map.

[0048] In one embodiment, if the image processing result is to determine the position of the target object; the first feature extraction module is specifically used to identify the initial feature map according to the regional level structure of the visual language large model, and determine the candidate calibration box corresponding to the target object in the task feature in the initial feature map;

[0049] According to the region-level structure of the visual language large model and the candidate calibration boxes in the initial feature map, each of the target feature maps is calibrated and mapped, and the initial feature map after the calibration box mapping is cropped according to the cropping structure to obtain unaligned regional features at each scale;

[0050] Based on the alignment structure, the unaligned regional features at each scale are scale-aligned to obtain a regional-level feature vector with the same feature dimension as the text encoding corresponding to the task text;

[0051] The region-level feature vector is matched with the text encoding of the target object in the task text to obtain the position of the target object.

[0052] In one embodiment, the device further comprises:

[0053] A second acquisition module is used to acquire a sample image set, a category label corresponding to each of the sample images, a position label corresponding to each of the sample images in the sample image set, category information corresponding to each of the position labels, and an initial visual language large model to be trained;

[0054] An embedding module, used for performing data processing on each of the sample images and the category annotations of each of the sample images according to the initial visual language large model to obtain a first feature map corresponding to the sample image and a category embedding corresponding to each of the category annotations;

[0055] A second convolution module is used to perform a multi-scale convolution transformation on the initial feature map based on the multi-scale structure of the initial visual language large model to obtain second feature maps of multiple scales, and perform region-level feature extraction on each of the second feature maps according to the region-level structure of the initial visual language large model to obtain a target region-level feature vector;

[0056] A training module is used to train the multi-scale structure and region-level structure of the initial visual language large model according to the region vector construction structure of the initial visual language large model, the category information, the category embedding, the target region-level feature vector and the region-level loss function construction structure to obtain a trained visual language large model.

[0057] In one of the embodiments, the region-level structure includes a cropping structure and an alignment structure; the second convolution module is specifically used to perform calibration frame mapping on each of the second feature maps according to the region-level structure of the initial visual language large model, and crop the second feature map after the calibration frame mapping according to the cropping structure to obtain the target unaligned region features at each scale;

[0058] The target unaligned region features at each scale are scale-aligned based on the alignment structure to obtain a target region-level feature vector with the same feature dimension as the category embedding.

[0059] In one of the embodiments, the alignment structure further includes a linear projection structure; the second convolution module is specifically used to perform spatial dimension maximum pooling calculation on the target unaligned region features at each scale according to the alignment structure to obtain an initial region-level feature vector;

[0060] Based on the linear projection structure, each of the initial region-level feature vectors is projected according to the dimension corresponding to the category code to obtain a target region-level feature vector.

[0061] In one embodiment, the training module is specifically used to encode the category information corresponding to each of the position annotations according to the regional vector construction structure of the initial visual language large model to obtain a category vector corresponding to each of the category information;

[0062] Based on the region-level loss function, a structure is constructed to calculate the loss value of the target region-level feature vector, the category embedding and the category vector at each scale to obtain the loss value at each scale;

[0063] A target loss value is determined according to the loss values ​​at each scale, and the multi-scale structure and region-level structure of the initial visual language large model are trained based on the target loss value until the target loss value meets the preset loss condition, thereby obtaining a trained visual language large model.

[0064] In one embodiment, the device further comprises:

[0065] A third acquisition module is used to acquire the image to be processed and the task text corresponding to the image to be processed based on the image processing task;

[0066] A second feature extraction module is used to extract features of the image to be processed and the task text according to the backbone network of the visual language large model to obtain an initial feature map corresponding to the image to be processed;

[0067] An encoding module, used for encoding the task text according to a text encoder of a visual language large model to obtain a text encoding;

[0068] A determination module is used to determine the task feature corresponding to the task text based on the text encoding.

[0069] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0070] Based on the image processing task, obtaining an initial feature map of the image to be processed and determining the task features of the image processing task;

[0071] Performing multi-scale convolution on the initial feature map based on the multi-scale structure of the visual language large model to obtain target feature maps of multiple scales;

[0072] Performing regional-level feature extraction on each of the target feature maps according to the regional-level structure of the visual language large model to obtain a regional-level feature vector;

[0073] The image to be processed is subjected to image processing based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature.

[0074] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0075] Based on the image processing task, obtaining an initial feature map of the image to be processed and determining the task features of the image processing task;

[0076] Performing multi-scale convolution on the initial feature map based on the multi-scale structure of the visual language large model to obtain target feature maps of multiple scales;

[0077] Performing regional-level feature extraction on each of the target feature maps according to the regional-level structure of the visual language large model to obtain a regional-level feature vector;

[0078] The image to be processed is subjected to image processing based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature.

[0079] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0080] Based on the image processing task, obtaining an initial feature map of the image to be processed and determining the task features of the image processing task;

[0081] Performing multi-scale convolution on the initial feature map based on the multi-scale structure of the visual language large model to obtain target feature maps of multiple scales;

[0082] Performing regional-level feature extraction on each of the target feature maps according to the regional-level structure of the visual language large model to obtain a regional-level feature vector;

[0083] The image to be processed is subjected to image processing based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature.

[0084] The above-mentioned image processing method, apparatus, computer equipment, computer-readable storage medium and computer program product, through the multi-scale structure to convolve the image to be processed, the model can capture the features of the image to be processed at different scales, so as to more comprehensively understand the image content, and further extract the features of the target feature map through the region-level structure to obtain the region-level feature vector, so that the visual language large model can more accurately focus on the key areas in the image to be processed, extract the local features closely related to the task text, improve the visual language large model for the position perception of objects of different sizes, and thus can flexibly adapt to different types of visual tasks, can effectively improve the accuracy of the visual language large model in dense prediction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0086] Figure 1 is a schematic flow chart of an image processing method in one embodiment;

[0087] Figure 2 A schematic diagram of a process for determining a multi-scale feature map in one embodiment;

[0088] Figure 3 is a schematic diagram of a multi-scale structure in one embodiment;

[0089] Figure 4 is a schematic diagram of a downsampling structure in one embodiment;

[0090] Figure 5 A schematic diagram of a process of performing task processing on a regional level structure in one embodiment;

[0091] Figure 6 A schematic diagram of a process for training an initial visual language large model in one embodiment;

[0092] Figure 7 A schematic diagram of a flow chart of embedding processing performed by a text encoder in one embodiment;

[0093] Figure 8 A schematic flow chart of data processing steps for a region-level structure in one embodiment;

[0094] Fig. 9 is a schematic diagram of a regional level structure in one embodiment;

[0095] Fig.10 A schematic diagram of a flow chart of data processing for an alignment structure in one embodiment;

[0096] Fig.11 is a schematic diagram of a linear projection structure in one embodiment;

[0097] Fig.12 A schematic diagram of a process of calculating the loss of an initial visual language large model in one embodiment;

[0098] Fig.13 A schematic diagram of one-hot encoding of a region vector construction structure in one embodiment;

[0099] Fig.14 A schematic diagram of a region-level loss function structure in one embodiment;

[0100] Fig.15 A schematic diagram of a model architecture of a visual language large model in one embodiment;

[0101] Fig.16 A schematic diagram of an example flow of obtaining an initial feature map and task features in another embodiment;

[0102] Fig.17 A schematic diagram of a process for obtaining an initial feature map in one embodiment;

[0103] Fig.18 is a structural block diagram of an image processing device in one embodiment;

[0104] Fig.19 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0105] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0106] In traditional technology, the backbone network of the large visual language model has a large receptive field for feature extraction and feature analysis of the input image, and the image feature analysis scale of the large model is relatively single. Therefore, the traditional visual language can only analyze the global features of the input image, resulting in poor ability to capture local details and lack of perception of local areas, which makes the traditional large visual language model poor in zero-sample generalization in dense prediction tasks, such as, but not limited to, target detection, image segmentation and other tasks. Therefore, in order to improve the position perception ability of the large visual language model for local areas and enable it to be aware of local area information, this application proposes a new image processing method.

[0107] In one embodiment, Figure 1 As shown, an image processing method is provided. This embodiment takes the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0108] Step 102: Based on the image processing task, an initial feature map of the image to be processed is obtained and a task feature of the image processing task is determined.

[0109] In the embodiment of the present application, the terminal responds to the user's triggering operation on the image processing task and receives the image to be processed input by the user. First, the terminal sends the image to be processed into the visual language model, and the image encoder based on the visual language model obtains the initial feature map of the image to be processed. At the same time, the terminal determines the task features according to the specific processing task. For example, if the image processing task is to identify a specific object in the image, then the task features include the attribute, classification and other feature information of the object.

[0110] Step 104, performing multi-scale convolution on the initial feature map based on the multi-scale structure of the visual language large model to obtain target feature maps of multiple scales.

[0111] Among them, the visual language large model is improved on the basis of the CLIP model architecture, adding multi-scale structure and regional level structure to obtain the visual language large model.

[0112] In an embodiment of the present application, the terminal performs a multi-scale convolution operation on the initial feature map according to the convolution kernels of different sizes (for example, convolution kernels of different sizes such as 3x3, 5x5, etc.) in the multi-scale structure of the large visual language model. The convolution kernel of each scale moves in the space of the initial feature map, thereby extracting the local structural information of the image to be processed, and extracting features from the initial feature map according to different levels and different ranges to obtain target feature maps at multiple scales. Each convolution operation generates a new target feature map, in which multiple target feature maps of different scales contain image features captured at different scales. The target feature map helps the model understand the different levels of features of the image, including various information from details to the global, so that the large visual language model can have a better perception effect on the global information and local information of the image to be processed.

[0113] Step 106, extracting regional features from each target feature map according to the regional structure of the visual language model to obtain a regional feature vector.

[0114] In an embodiment of the present application, the terminal searches for an area of ​​interest (for example, the location of an object) on a target feature map of each scale through the region-level structure of a large visual language model. The area of ​​interest may be determined by a sliding window or a region proposal network, and region-level features are extracted from each region according to the information in the task features. Region-level feature extraction is performed on target feature maps of different scales. Specifically, region-level feature extraction includes cropping features in each region of interest, and aligning these cropped features to a uniform size through ROI Pooling (region of interest pooling) and ROI Align (region of interest alignment) operations to ensure that regions of different sizes can generate feature representations with uniform dimensions, thereby obtaining region-level feature vectors extracted from target feature maps of each scale.

[0115] Step 108 , performing image processing on the image to be processed based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature.

[0116] In an embodiment of the present application, the terminal uses region-level feature vectors and task features to process image data to obtain an image processing result corresponding to the final task feature. Specifically, the visual language model also includes a post-processing module, for example, a fully connected network, to perform tasks such as classification, segmentation, and detection. The post-processing module uses region-level feature vectors to predict the output of the task, thereby obtaining an image processing result that meets user needs.

[0117] In the above image processing method, the image to be processed is convolved through a multi-scale structure, and the model can capture the features of the image to be processed at different scales, so as to understand the image content more comprehensively. The target feature map is further extracted through the region-level structure to obtain the region-level feature vector, so that the visual language model can focus more accurately on the key areas in the image to be processed and extract local features closely related to the task text, thereby improving the visual language model's ability to perceive the position of objects of different sizes, and thus can flexibly adapt to different types of visual tasks, which can effectively improve the accuracy of the visual language model in dense prediction tasks.

[0118] In an exemplary embodiment, the multi-scale structure includes an initial convolution structure, a first downsampling structure, and a second downsampling structure, and the target feature map includes a first scale feature map, a second scale feature map, and a third scale feature map; Figure 2 As shown, step 104 includes steps 202 to 206. Among them:

[0119] Step 202: Perform a convolution transformation on the initial feature map according to the initial convolution structure to obtain a first scale feature map.

[0120] In the present application embodiment, Figure 3 As shown in the figure, the initial convolution structure can be a 1x1 convolution kernel. The terminal performs a convolution transformation on the initial feature map according to the channel relationship between features of the initial feature map learned by the convolution kernel during the training process (that is, the weighted combination between different channels to generate a new feature representation), extracts high-level abstract features in the initial feature map, and then obtains a first-scale feature map (first-scale feature map) at the first scale.

[0121] Step 204: Sampling and distribution correction are performed on the first scale feature map according to the first down-sampling structure to obtain a second scale feature map.

[0122] In the present application embodiment, Figure 3 As shown, the terminal inputs the first scale feature map output by the initial convolution structure into the first downsampling module. Specifically, as Figure 4As shown in the figure, the first downsampling structure includes a 2x2 convolution kernel, a batch normalization structure and an activation function (ReLU, an activation function). The first downsampling structure first uses a 2x2 convolution kernel to downsample the first-scale feature map, and then performs distribution correction on the downsampled first-scale feature map according to the batch normalization structure. Finally, the activation function ReLU is used to improve the expression ability of the first-scale feature map to obtain the second-scale feature map.

[0123] Step 206: Sampling and distribution correction are performed on the second scale feature map according to the second down-sampling structure to obtain a third scale feature map.

[0124] In the embodiment of the present application, the terminal samples and corrects the distribution of the second scale feature map according to the second downsampling structure in accordance with the same principle as step 204 to obtain a third scale feature map, wherein the detailed process of sampling and distribution correction is not repeated in this embodiment.

[0125] In this embodiment, the initial feature map is compressed and abstractly represented through the convolution transformation of the initial convolution structure, and then sampling and distribution correction are performed according to the first downsampling structure and the second downsampling structure, so that more high-level feature information can be extracted, so that the visual language large model can extract target feature maps of different scales from the image to be processed, and obtain feature representations at different levels of abstraction, thereby improving the comprehensiveness of the visual language large model in feature extraction of the image to be processed and improving the accuracy of subsequent image processing.

[0126] In an exemplary embodiment, Figure 5 As shown, if the image processing result is to determine the position of the target object, step 106 includes steps 502 to 508. Among them:

[0127] Step 502, identifying the initial feature map according to the region-level structure of the visual language model, and determining the candidate calibration box corresponding to the target object in the task feature in each target feature map.

[0128] In an embodiment of the present application, first, the terminal extracts feature representations related to the target object from the initial feature map based on the visual language large model. The terminal uses the regional level structure of the visual language large model to identify the target object and generate a calibration frame on the initial feature map, and performs preliminary identification over a larger range in the initial feature map to obtain multiple candidate calibration frames related to the target object in the task features.

[0129] Step 504, mapping the calibration frames of each target feature map according to the region-level structure of the visual language model and the candidate calibration frames in the initial feature map, and cropping the initial feature map after the calibration frame mapping according to the cropping structure to obtain the unaligned region features at each scale.

[0130] In an embodiment of the present application, the terminal maps each candidate calibration frame to the same scale as each target feature map, that is, obtains target feature maps after multiple calibration frames are mapped. Then, the terminal crops out the high-quality feature map corresponding to each calibration frame from the target feature map after the calibration frame is mapped according to the cropping structure in the region-level structure. Since different calibration frames may correspond to different scales, the cropped target feature map retains the difference in scale. Therefore, after the terminal performs calibration frame mapping on the target feature map, the unaligned region features at each scale are obtained.

[0131] Step 506: align the unaligned regional features at each scale based on the alignment structure to obtain a regional level feature vector having the same feature dimension as the text encoding corresponding to the task text.

[0132] In an embodiment of the present application, the terminal performs a scale alignment operation on the cropped unaligned region features through an alignment structure, maps the feature map pixels of the cropped unaligned region features to a fixed spatial size according to the text encoding corresponding to the target object in the task text, and obtains a region-level feature vector with the same feature dimension as the text encoding of the target object.

[0133] Step 508: perform similarity matching between the region-level feature vector and the text encoding of the target object in the task text to obtain the position of the target object.

[0134] In the embodiment of the present application, the terminal performs similarity matching between the region-level feature vectors aligned at multiple scales and the embedded encoding of the target object in the task text by calculating the cosine similarity or other similarity metrics between the vectors. The terminal identifies the candidate calibration box with the highest matching score as the location of the target object, and the location information can be used for further image processing tasks, such as target detection, object recognition, etc.

[0135] In this embodiment, candidate calibration boxes are generated through region-level structures for final similarity matching. The region-level structure is combined with visual features and language features to enable the large visual language model to understand and process the image to be processed at the global and local semantic levels, achieve local and accurate determination of the target object position, and improve the accuracy of the large visual language model in dense prediction tasks.

[0136] In an exemplary embodiment, Figure 6 As shown, before step 102, the visual language model needs to be trained, and the method also includes steps 602 to 608. Among them:

[0137] Step 602, obtaining a sample image set, a category label corresponding to each sample image, a position label corresponding to each sample image in the sample image set, category information corresponding to each position label, and an initial visual language large model to be trained.

[0138] In the embodiment of the present application, the sample image set can be obtained by mixing the target detection dataset and the image segmentation dataset. Specifically, the target detection dataset can be a coco detection dataset or a dataset collected for a specific business scenario; the target detection dataset can be represented as ,in represents the i-th image in the target detection dataset, Represents the coordinates of multiple bounding boxes corresponding to the i-th image , that is, the position annotation corresponding to the sample image, Indicates the category information of the i-th image ; The category name of each target detection dataset is recorded as , that is, the category label corresponding to each sample image.

[0139] The image segmentation dataset can be the ade20k dataset or a dataset collected in a business scenario. The image segmentation dataset can be expressed as ,in represents the i-th image in the dataset, Represents multiple masks corresponding to the i-th image, and converts the mask into an external matrix, expressed as , that is, the position annotation corresponding to the sample image, Indicates the category information of the i-th image ; The category name of each image segmentation dataset is recorded as , that is, the category label corresponding to each sample image.

[0140] After mixing the target detection dataset and the image segmentation dataset, a mixed dataset is obtained, that is, a sample image set is obtained, where the mixed dataset is expressed as ,in Represents the total number of images in the target detection dataset, Represents the total number of images in the image segmentation dataset; the category name of the mixed dataset is recorded as , the total amount is recorded as n.

[0141] Step 604 , performing data processing on each sample image and the category annotation of each sample image according to the initial visual language large model, to obtain a first feature map corresponding to the sample image and a category embedding corresponding to each category annotation.

[0142] In the embodiment of the present application, first, the terminal freezes the pre-trained weights of the initial visual language model, which are the weights of each part in the original structure of the visual language model. Then, fine-tuning training is performed on the multi-scale structure and regional structure of the initial visual language model. The terminal inputs each sample image in the sample image set into the backbone network of the visual language model, and obtains the first feature map corresponding to each sample image through the backbone network, which is recorded as . For category annotation, such as Figure 7 As shown in , the terminal inputs the category label (i.e., category name) of each sample image into the text encoder, and obtains the category embedding corresponding to the category label through the text encoder, which is recorded as .

[0143] Step 606, based on the multi-scale structure of the initial visual language large model, a multi-scale convolution transformation is performed on the initial feature map to obtain second feature maps of multiple scales, and region-level feature extraction is performed on each second feature map according to the region-level structure of the initial visual language large model to obtain a target region-level feature vector.

[0144] In an embodiment of the present application, first, the terminal performs a convolution transformation on the first feature map through the multi-scale structure in the initial visual language large model to generate a second feature map of multiple scales, and the first feature map captures different visual details and semantic information of the sample image at different scales. Then, based on the regional level structure of the initial visual language large model, the terminal performs regional level feature extraction on the second feature map at each scale, extracts the target regional level feature map from the pyramid feature map, for example, through object detection or region proposal network, extracts the target regional level feature vector from the second feature map, and obtains the target regional level feature vector that contains the semantic information and position information of each calibration box in the local semantics.

[0145] Step 608, training the multi-scale structure and region-level structure of the initial visual language large model according to the region vector construction structure, category information, category embedding, target region-level feature vector and region-level loss function construction structure of the initial visual language large model to obtain a trained visual language large model.

[0146] In an embodiment of the present application, the terminal constructs a loss function and calculates the loss value for the category information, category embedding and target region-level feature vector according to the region vector construction structure and the region-level loss function construction structure, and fine-tunes the model parameters of the initial visual language large model according to the stochastic gradient descent algorithm (SGD) or the adaptive learning rate optimization algorithm (Adam) to obtain a trained visual language large model.

[0147] In this embodiment, through multi-scale convolution transformation and region-level feature extraction, it is possible to capture detailed information of the image at different scales, and align this information with the text embedding, so that the large visual language model can effectively recognize different scales and positions, and improve the position perception ability of objects of different sizes.

[0148] In an exemplary embodiment, the region-level structure includes a clipping structure and an alignment structure, and the alignment structure further includes a linear projection structure; Figure 8 As shown, step 606 includes steps 802 to 804. Among them:

[0149] Step 802, calibrate and map the second feature maps according to the region-level structure of the initial visual language model, and crop the second feature maps after the calibration frame mapping according to the cropping structure to obtain the target unaligned region features at each scale.

[0150] In the embodiment of the present application, the terminal follows the same principle as in step 504, such as Fig. 9 As shown, through the regional level structure of the initial visual language model, the second feature map at different scales is calibrated and cropped according to the position annotation corresponding to the sample image to obtain the target misaligned regional features. The specific process will not be repeated in this embodiment. Take three scales as an example to illustrate, map the bounding box coordinates (position annotation) corresponding to the sample image to the corresponding positions of the feature map of the first scale, the feature map of the second scale, and the feature map of the third scale, that is, ,in , , , I represents the total number of regions of interest contained in each image. If it is a target detection dataset, I=B, and if it is an image segmentation dataset, I=M. The terminal crops the second feature maps of the three scales after the calibration frame is mapped according to the cropping structure, and obtains the unaligned region features corresponding to each bounding box at each scale, which are recorded as .

[0151] Step 804: align the target unaligned region features at each scale based on the alignment structure to obtain a target region-level feature vector with the same feature dimension as the category embedding.

[0152] In the embodiment of the present application, the terminal follows the same principle as in step 506, such as Fig. 9 As shown, through the alignment structure in the region-level structure, the target unaligned region features at each scale are scale-aligned to obtain a target region-level feature vector with the same feature dimension as the category embedding. The specific process will not be repeated in this embodiment.

[0153] In this embodiment, the calibration box mapping and cropping operations are used to ensure that the features of the marked area of ​​the second feature map not only contain the semantic information of the target object, but also retain its spatial position and scale information in the sample image. The scale alignment operation enables the large visual language model to process input images of different scales and resolutions while maintaining the consistency and comparability of the features, thereby improving the generalization ability of the large visual language model in practical applications and improving the accuracy of image processing.

[0154] In an exemplary embodiment, the alignment structure also includes a linear projection structure; Fig.10 As shown, step 804 includes steps 1002 to 1004. Among them:

[0155] Step 1002 , performing spatial dimension maximum pooling calculation on target unaligned region features at each scale according to the alignment structure to obtain an initial region-level feature vector.

[0156] In an embodiment of the present application, the terminal performs a maximum pooling calculation in the spatial dimension for the target unaligned region features at each scale according to the alignment structure of the initial visual language large model to obtain a one-dimensional feature vector, that is, an initial region-level feature vector corresponding to each target unaligned region feature.

[0157] Step 1004 , based on the linear projection structure, project each initial region-level feature vector according to the dimension corresponding to the category code to obtain a target region-level feature vector.

[0158] In the present application embodiment, Fig.11 As shown, the terminal inputs the one-dimensional feature vector (initial region-level feature vector) corresponding to each target unaligned region feature into the linear projection structure, and linearly projects the one-dimensional feature vector according to the feature dimension of the category embedding based on the linear projection structure, so as to ensure that the initial region-level feature vector of each target region can match the category embedding or other specific dimension input required for subsequent tasks, and obtain the target one-dimensional feature vector with the same feature dimension as the category embedding, that is, the target region-level feature vector.

[0159] In this embodiment, the target unaligned region features and category embeddings are aligned through the alignment structure and linear projection structure of the visual language large model, so that the visual language large model can effectively process the target region information extracted from the image and convert it into a feature representation suitable for processing by various parts of the visual language large model, thereby improving the performance and generalization capabilities of the visual language large model in complex vision and language tasks.

[0160] In an exemplary embodiment, Fig.12 As shown, step 608 includes steps 1202 to 1206. Among them:

[0161] Step 1202 , encoding the category information corresponding to each position label according to the regional vector construction structure of the initial visual language large model, and obtaining the category vector corresponding to each category information.

[0162] In the present application embodiment, Fig.13 As shown, the terminal inputs the category information corresponding to each position annotation contained in each sample image into the regional vector construction structure of the initial visual language large model, wherein each position annotation contains multiple coordinates, each coordinate has its corresponding category index, that is, each position annotation has corresponding category information. For each position annotation containing the index of the coordinate, the terminal performs one hot encoding on it, and then obtains the category vector corresponding to each category information.

[0163] Step 1204 , based on the region-level loss function, a structure is constructed to calculate the loss value of the target region-level feature vector, category embedding, and category vector at each scale to obtain the loss value at each scale.

[0164] In the embodiment of the present application, the terminal inputs the target region-level feature vector at each scale, the category embedding corresponding to the target region-level feature vector, and the category vector into the region-level loss function construction structure, and obtains the loss value of each scale through the region-level loss function construction structure. Specifically, Fig.14 As shown, for each target region-level feature vector at each scale , respectively, and calculate the similarity value with each category embedding, taking cosine similarity as an example, the similarity result is recorded as Among them, i=1,...,3,j=1,...,I,k=1,...,n,k is the category index (category information); then the terminal uses the softmax operation to convert the above similarity results into probability distribution along the dimension of category information k, which is recorded as ; Finally, the terminal is based on the probability distribution and the category vector , using cross entropy loss, we get the region-level loss function, denoted as , where i=1,...,3, j=1,...,I, then the terminal calculates the loss value based on each scale , sum along the j dimension to get the loss value at each scale, recorded as .

[0165] Step 1206, determine the target loss value according to the loss values ​​at each scale, and train the multi-scale structure and region-level structure of the initial visual language model based on the target loss value until the target loss value meets the preset loss condition, thereby obtaining a trained visual language model.

[0166] In the embodiment of the present application, the terminal sums the loss values ​​of each scale to obtain the final target loss value, and fine-tunes the multi-scale structure and regional structure of the initial visual language model based on the stochastic gradient descent algorithm or the adaptive learning rate optimization algorithm until the target loss value meets the preset loss condition, and the trained visual language model is completed. The preset loss condition can be that the target loss value is lower than the preset loss threshold, or the number of iterations meets the requirements of the preset iteration rounds.

[0167] In this embodiment, by integrating the loss values ​​at different scales, the initial visual language large model can learn more generalized feature representations, and construct the probability distribution of feature similarity and category prediction through the target area-level feature vector, category embedding and category vector to guide the training of the initial visual language large model. The classification accuracy and category prediction ability at each scale can be accurately evaluated during the training process, thereby optimizing the feature learning and generalization ability of the initial visual language large model and improving the accuracy of the visual language large model in dense prediction tasks.

[0168] In a specific embodiment, an example of a visual language large model training method is provided, such as Fig.15 As shown, the method includes:

[0169] Step 1501, obtaining a first feature map (feature map) and a position annotation (gt box) of each sample image in a sample image set;

[0170] Step 1502, processing the first feature map according to the multi-scale structure to obtain a second feature map, and processing the second feature map and the sample label based on the region-level structure to obtain a target region-level feature vector (region feature);

[0171] Step 1503, encoding the category label of each sample image according to the text encoder to obtain a category embedding;

[0172] Step 1504, performing one-hot encoding on the category information corresponding to the location annotation according to the region vector construction structure to obtain a category vector;

[0173] Step 1505, construct a structure based on the region-level loss function to perform data processing on the target region-level feature vector, category embedding and category vector to obtain the target loss value of the initial visual language large model, and train the multi-scale structure and region-level structure of the initial visual language large model based on the target loss value.

[0174] In an exemplary embodiment, Fig.16 As shown, step 102 includes steps 1602 to 1606. Among them:

[0175] Step 1602: Based on the image processing task, obtain the image to be processed and the task text corresponding to the image to be processed.

[0176] In an embodiment of the present application, the terminal obtains an image to be processed and a corresponding task text from an actual application scenario, wherein the image is a visual input processed by a large visual language model input by a user, and the task text is a specific task or requirement to be achieved for image processing.

[0177] Step 1604, extract features of the image to be processed and the task text according to the backbone network of the visual language model to obtain an initial feature map corresponding to the image to be processed.

[0178] In the present application embodiment, Fig.17 As shown, the terminal uses the backbone network of the visual language large model to extract features of the image to be processed and obtains an initial feature map of the high-level abstract representation of the image to be processed.

[0179] Step 1606, encode the task text according to the text encoder of the visual language large model to obtain a text encoding.

[0180] In an embodiment of the present application, the terminal uses a text encoder of a large visual language model to encode the task text. The text encoder can convert the natural language task description into a vectorized representation that can be understood and processed by the large visual language model.

[0181] Step 1608, determining the task features corresponding to the task text based on the text encoding.

[0182] In an embodiment of the present application, the terminal extracts task-related features of the text encoding based on the text mapping relationship learned by pre-training of the large visual language model, performs semantic understanding of the task text, determines the user's intention, and obtains the task feature corresponding to the task text currently input by the user. The task feature can be a type of image processing, such as target detection, image segmentation, phrase localization, or regional-level image quality evaluation.

[0183] In this embodiment, by converting the image to be processed and the task text into corresponding feature representations, task features and initial feature maps are obtained, which provide a basis and prerequisite for the visual language model to effectively process visual and language information in subsequent tasks.

[0184] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0185] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0186] In an exemplary embodiment, Fig.18 As shown, an image processing device 1800 is provided, comprising: a first acquisition module 1801, a first convolution module 1802, a first feature extraction module 1803 and an image processing module 1804, wherein:

[0187] The first acquisition module 1801 is used to acquire an initial feature map of the image to be processed and determine the task features of the image processing task based on the image processing task;

[0188] A first convolution module 1802 is used to perform multi-scale convolution on the initial feature map based on the multi-scale structure of the visual language large model to obtain target feature maps of multiple scales;

[0189] The first feature extraction module 1803 is used to extract regional features from each target feature map according to the regional structure of the visual language model to obtain a regional feature vector;

[0190] The image processing module 1804 is used to perform image processing on the image to be processed based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature.

[0191] In one embodiment, the multi-scale structure includes an initial convolution structure, a first down-sampling structure, and a second down-sampling structure, and the target feature map includes a first scale feature map, a second scale feature map, and a third scale feature map; the first convolution module 1802 is specifically used to perform a convolution transformation on the initial feature map according to the initial convolution structure to obtain a first scale feature map;

[0192] Sampling and distribution correcting the first scale feature map according to the first down-sampling structure to obtain a second scale feature map;

[0193] The second scale feature map is sampled and distribution corrected according to the second down-sampling structure to obtain a third scale feature map.

[0194] In one embodiment, if the image processing result is to determine the position of the target object; the first feature extraction module 1803 is specifically used to identify the initial feature map according to the regional level structure of the visual language model, and determine the candidate calibration box corresponding to the target object in the task feature in the initial feature map;

[0195] According to the region-level structure of the visual language model and the candidate calibration boxes in the initial feature map, the calibration boxes of each target feature map are mapped, and the initial feature map after the calibration box mapping is cropped according to the cropping structure to obtain the unaligned regional features at each scale;

[0196] Based on the alignment structure, the unaligned regional features at each scale are scale-aligned to obtain a regional-level feature vector with the same feature dimension as the text encoding corresponding to the task text;

[0197] The region-level feature vector is similarly matched with the text encoding of the target object in the task text to obtain the location of the target object.

[0198] In one embodiment, the apparatus 1800 further includes:

[0199] The second acquisition module is used to acquire a sample image set, a category label corresponding to each sample image, a position label corresponding to each sample image in the sample image set, category information corresponding to each position label, and an initial visual language large model to be trained;

[0200] An embedding module, used to perform data processing on each sample image and the category annotation of each sample image according to the initial visual language large model, to obtain a first feature map corresponding to the sample image and a category embedding corresponding to each category annotation;

[0201] A second convolution module is used to perform a multi-scale convolution transformation on the initial feature map based on the multi-scale structure of the initial visual language large model to obtain second feature maps of multiple scales, and perform regional-level feature extraction on each second feature map according to the regional-level structure of the initial visual language large model to obtain a target regional-level feature vector;

[0202] The training module is used to train the multi-scale structure and region-level structure of the initial visual language model according to the region vector construction structure, category information, category embedding, target region-level feature vector and region-level loss function construction structure of the initial visual language model to obtain a trained visual language model.

[0203] In one embodiment, the region-level structure includes a cropping structure and an alignment structure; the second convolution module is specifically used to perform calibration frame mapping on each second feature map according to the region-level structure of the initial visual language large model, and to crop the second feature map after the calibration frame mapping according to the cropping structure to obtain the target unaligned region features at each scale;

[0204] Based on the alignment structure, the unaligned target region features at each scale are scale-aligned to obtain a target region-level feature vector with the same feature dimension as the category embedding.

[0205] In one of the embodiments, the alignment structure further includes a linear projection structure; the second convolution module is specifically used to perform spatial dimension maximum pooling calculation on the target unaligned region features at each scale according to the alignment structure to obtain an initial region-level feature vector;

[0206] Based on the linear projection structure, each initial region-level feature vector is projected according to the dimension corresponding to the category code to obtain the target region-level feature vector.

[0207] In one embodiment, the training module is specifically used to encode the category information corresponding to each position label according to the regional vector construction structure of the initial visual language large model, and obtain the category vector corresponding to each category information;

[0208] Based on the regional loss function, a structure is constructed to calculate the loss value of the target regional feature vector, category embedding and category vector at each scale to obtain the loss value at each scale;

[0209] The target loss value is determined according to the loss values ​​at each scale, and the multi-scale structure and region-level structure of the initial visual language model are trained based on the target loss value until the target loss value meets the preset loss condition, thereby obtaining a trained visual language model.

[0210] In one embodiment, the apparatus 1800 further includes:

[0211] A third acquisition module is used to acquire the image to be processed and the task text corresponding to the image to be processed based on the image processing task;

[0212] The second feature extraction module is used to extract features of the image to be processed and the task text according to the backbone network of the visual language model to obtain an initial feature map corresponding to the image to be processed;

[0213] An encoding module, used for encoding the task text according to the text encoder of the visual language model to obtain a text encoding;

[0214] The determination module is used to determine the task features corresponding to the task text based on the text encoding.

[0215] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0216] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.19 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store images to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0217] Those skilled in the art will understand that Fig.19 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0218] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0219] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0220] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0221] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0222] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0223] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0224] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Based on the image processing task, obtaining an initial feature map of the image to be processed and determining the task features of the image processing task; Performing multi-scale convolution on the initial feature map based on the multi-scale structure of the visual language large model to obtain target feature maps of multiple scales; Performing regional-level feature extraction on each of the target feature maps according to the regional-level structure of the visual language large model to obtain a regional-level feature vector; The image to be processed is subjected to image processing based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature.

2. The method according to claim 1, characterized in that The multi-scale structure includes an initial convolution structure, a first downsampling structure and a second downsampling structure, and the target feature map includes a first-scale feature map, a second-scale feature map and a third-scale feature map; the multi-scale structure based on the visual language large model performs multi-scale convolution on the initial feature map to obtain target feature maps of multiple scales, including: Performing a convolution transformation on the initial feature map according to the initial convolution structure to obtain the first scale feature map; Sampling and distribution correcting the first scale feature map according to the first down-sampling structure to obtain the second scale feature map; The second scale feature map is sampled and distribution corrected according to the second down-sampling structure to obtain the third scale feature map.

3. The method according to claim 1, characterized in that If the image processing result is to determine the position of the target object; performing regional-level feature extraction on each of the target feature maps according to the regional-level structure of the visual language large model to obtain a regional-level feature vector includes: Identify the initial feature map according to the region-level structure of the visual language large model, and determine a candidate calibration box corresponding to the target object in the task feature in the initial feature map; According to the region-level structure of the visual language large model and the candidate calibration boxes in the initial feature map, each of the target feature maps is calibrated and mapped, and the initial feature map after the calibration box mapping is cropped according to the cropping structure to obtain unaligned regional features at each scale; Based on the alignment structure, the unaligned regional features at each scale are scale-aligned to obtain a regional-level feature vector with the same feature dimension as the text encoding corresponding to the task text; The region-level feature vector is matched with the text encoding of the target object in the task text to obtain the position of the target object.

4. The method according to claim 1, characterized in that Before obtaining the initial feature map of the image to be processed and determining the task features of the image processing task based on the image processing task, the method further includes: Obtaining a sample image set, a category label corresponding to each of the sample images, a position label corresponding to each of the sample images in the sample image set, category information corresponding to each of the position labels, and an initial visual language large model to be trained; Performing data processing on each of the sample images and the category annotations of each of the sample images according to the initial visual language large model to obtain a first feature map corresponding to the sample image and a category embedding corresponding to each of the category annotations; Based on the multi-scale structure of the initial visual language large model, the initial feature map is subjected to a multi-scale convolution transformation to obtain second feature maps of multiple scales, and according to the regional level structure of the initial visual language large model, regional level features are extracted from each of the second feature maps to obtain a target regional level feature vector; The multi-scale structure and region-level structure of the initial visual language large model are trained according to the region vector construction structure of the initial visual language large model, the category information, the category embedding, the target region-level feature vector and the region-level loss function construction structure to obtain a trained visual language large model.

5. The method according to claim 4, characterized in that The region-level structure includes a cropping structure and an alignment structure; the step of extracting region-level features from each of the second feature maps according to the region-level structure of the initial visual language large model to obtain a target region-level feature vector includes: Performing calibration frame mapping on each of the second feature maps according to the region-level structure of the initial visual language large model, and cropping the second feature maps after the calibration frame mapping according to the cropping structure to obtain target misaligned region features at each scale; The target unaligned region features at each scale are scale-aligned based on the alignment structure to obtain a target region-level feature vector with the same feature dimension as the category embedding.

6. The method according to claim 5, characterized in that The alignment structure further includes a linear projection structure; and the scale alignment of the target unaligned region features at each scale based on the alignment structure to obtain a target region-level feature vector with the same feature dimension as the category embedding includes: Performing spatial dimension maximum pooling calculation on the target unaligned region features at each scale according to the alignment structure to obtain an initial region-level feature vector; Based on the linear projection structure, each of the initial region-level feature vectors is projected according to the dimension corresponding to the category code to obtain a target region-level feature vector.

7. The method according to claim 4, characterized in that The multi-scale structure and the regional level structure of the initial visual language large model are trained according to the regional vector construction structure of the initial visual language large model, the category information, the category embedding, the target region-level feature vector and the region-level loss function construction structure to obtain the trained visual language large model, including: Encoding the category information corresponding to each of the position annotations according to the regional vector construction structure of the initial visual language large model to obtain a category vector corresponding to each of the category information; Based on the region-level loss function, a structure is constructed to calculate the loss value of the target region-level feature vector, the category embedding and the category vector at each scale to obtain the loss value at each scale; A target loss value is determined according to the loss values ​​at each scale, and the multi-scale structure and region-level structure of the initial visual language large model are trained based on the target loss value until the target loss value meets the preset loss condition, thereby obtaining a trained visual language large model.

8. The method according to claim 1, characterized in that: The step of obtaining an initial feature map of the image to be processed and determining the task features of the image processing task based on the image processing task includes: Based on the image processing task, obtaining an image to be processed and a task text corresponding to the image to be processed; Extract features of the image to be processed and the task text according to the backbone network of the visual language large model to obtain an initial feature map corresponding to the image to be processed; Encoding the task text according to the text encoder of the visual language large model to obtain a text encoding; A task feature corresponding to the task text is determined based on the text encoding.

9. An image processing device, characterized in that: The device comprises: A first acquisition module, used to acquire an initial feature map of an image to be processed and determine a task feature of the image processing task based on an image processing task; A first convolution module, configured to perform multi-scale convolution on the initial feature map based on the multi-scale structure of the visual language large model to obtain target feature maps of multiple scales; A first feature extraction module is used to extract regional features from each of the target feature maps according to the regional structure of the visual language model to obtain a regional feature vector; The image processing module is used to perform image processing on the image to be processed based on the region-level feature vector and the task feature to obtain an image processing result corresponding to the task feature.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images