Image processing method and device based on multi-modal large model, equipment and medium

Abstract

The invention provides an image processing method and device based on a multi-modal large model, equipment and a medium, and the method comprises the steps: obtaining a to-be-processed image and problem description information for the to-be-processed image; performing down-sampling processing and segmentation processing on the to-be-processed image to obtain a down-sampling feature map and a plurality of segmentation feature maps of a multi-feature extraction angle; encoding the down-sampling feature map of the multi-feature extraction angle and the plurality of segmentation feature maps through a visual encoder to obtain a down-sampling encoding feature and a segmentation encoding feature; and fusing the down-sampling coding feature and the segmentation coding feature to obtain a fused feature, and processing the fused feature and the problem description information through a large language model to obtain a processing result.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to an image processing method, apparatus, device, and medium based on a multimodal large model. Background Technology

[0002] With the rise of generative large language models, text-generative multimodal large models have gradually come into view. Compared with traditional discriminative multimodal large models, text-generative multimodal large models, represented by LLAVA, have rapidly gained widespread recognition due to their convenient interaction, universality of capabilities, and good scalability.

[0003] In the conventional multimodal large models mentioned above, without increasing the resolution, most multimodal large models have very poor perception ability for large images. This directly leads to severe model illusions when there is a lot of text or small objects in the image, making it impossible to give reliable results. Summary of the Invention

[0004] This disclosure provides at least one image processing method, apparatus, device, and medium based on a multimodal large model.

[0005] In a first aspect, embodiments of this disclosure provide an image processing method based on a multimodal large model, including:

[0006] Obtain the image to be processed and the problem description information for the image to be processed;

[0007] The image to be processed is subjected to downsampling and segmentation processing respectively to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles;

[0008] The downsampled feature map of the multi-feature extraction angle and the multiple segmented feature maps are encoded by a visual encoder to obtain downsampled encoded features and segmented encoded features.

[0009] The downsampling coding features and the segmentation coding features are fused to obtain fused features. The fused features and the problem description information are then processed using a large language model to obtain the processing result.

[0010] In one optional implementation, the step of performing downsampling and segmentation processing on the image to be processed to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles includes: performing downsampling and segmentation processing on the image to be processed through a dual-path visual input module to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles.

[0011] The step of fusing the downsampled coding features and the segmented coding features to obtain fused features includes: fusing the downsampled coding features and the segmented coding features through a dual-path feature fusion module to obtain the fused features.

[0012] In one optional implementation, the step of downsampling the image to be processed to obtain downsampled feature maps with multiple feature extraction angles includes:

[0013] The image to be processed is downsampled to obtain a downsampled image with multiple feature extraction angles;

[0014] The size of the downsampled image for each of the multi-feature extraction angles is normalized.

[0015] Based on the downsampled image after normalization, the downsampled feature map of the multi-feature extraction angle is determined.

[0016] In one optional implementation, determining the downsampled feature map of the multi-feature extraction angle based on the normalized downsampled image includes:

[0017] Based on the number of feature channels in the downsampled image from the multi-feature extraction angle, the downsampled image from the multi-feature extraction angle is segmented into feature maps to obtain multiple feature map patches; wherein, the number of the multiple feature map patches is the same as the number of multi-feature extraction angles;

[0018] The multiple feature map blocks are concatenated to obtain the downsampled feature map.

[0019] In one optional implementation, the step of segmenting the image to be processed to obtain multiple segmentation feature maps includes:

[0020] The image to be processed is divided into multiple sub-images; wherein the number of the multiple sub-images is the same as the number of the multi-feature extraction angles;

[0021] The multiple sub-images are stitched together to obtain the multiple segmentation feature maps.

[0022] In one optional implementation, the step of encoding the downsampled feature map of the multi-feature extraction angle and the multiple segmented feature maps using a visual encoder to obtain downsampled encoded features and segmented encoded features includes:

[0023] The downsampled feature map from the multiple feature extraction angles and the multiple segmented feature maps are fused to obtain the fused features;

[0024] The fused features are input into the visual encoder for processing to obtain output features; wherein, the output features include the downsampled coding features and the segmented coding features.

[0025] In one optional implementation, fusing the downsampled coding features and the segmented coding features to obtain fused features includes:

[0026] The downsampled encoded features are fused from the perspective of multi-feature extraction to obtain the first feature to be fused.

[0027] The segmented coding features are spatially fused from a spatial perspective to obtain a second feature to be fused.

[0028] The fusion feature is determined based on the first feature to be fused and the second feature to be fused.

[0029] In one optional implementation, the step of fusing the downsampled encoded features from the perspective of multi-feature extraction to obtain a first feature to be fused includes:

[0030] The downsampled encoded features are segmented to obtain the first sub-encoded features of each feature extraction angle in each feature channel;

[0031] The first sub-coded features are concatenated from the feature dimension of the downsampled encoded features to obtain the first feature to be fused.

[0032] In one optional implementation, the step of spatially fusing the segmented coding features from a spatial perspective to obtain a second feature to be fused includes:

[0033] The segmented coding features are divided to obtain second sub-coding features for each spatial angle;

[0034] The second sub-encoded feature is concatenated from the spatial dimension of the segmented encoded feature to obtain the second feature to be fused.

[0035] In one alternative implementation, the method further includes training a multimodal large model in the following manner:

[0036] Acquire training samples, and train the dual-path visual input module and the dual-path feature fusion module using the training samples to obtain the trained dual-path visual input module and the trained dual-path feature fusion module;

[0037] The training samples are used to train the trained dual-path visual input module, the trained dual-path feature fusion module, the visual encoder, and the large language model to obtain the multimodal large model.

[0038] Secondly, embodiments of this disclosure provide an image processing apparatus based on a multimodal large model, comprising:

[0039] The acquisition module is used to acquire the image to be processed and the problem description information for the image to be processed;

[0040] The dual-path visual input module is used to perform downsampling and segmentation processing on the image to be processed, respectively, to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles;

[0041] A visual encoder is used to encode the downsampled feature map of the multi-feature extraction angle and the multiple segmented feature maps to obtain downsampled encoded features and segmented encoded features.

[0042] The dual-path feature fusion module is used to fuse the downsampled coding features and the segmented coding features to obtain fused features, and then process the fused features and the problem description information through a large language model to obtain the processing result.

[0043] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0044] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible implementation of the first aspect.

[0045] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the first aspect of claim, or any possible implementation of the first aspect.

[0046] This disclosure provides an image processing method, apparatus, device, and medium based on a multimodal large model. In embodiments of this disclosure, firstly, an image to be processed and problem description information for the image to be processed are acquired; then, the image to be processed is subjected to downsampling and segmentation processing respectively to obtain downsampled feature maps from multiple feature extraction angles and multiple segmentation feature maps; next, the downsampled feature maps from multiple feature extraction angles and multiple segmentation feature maps are encoded by a visual encoder to obtain downsampled encoded features and segmentation encoded features; finally, the downsampled encoded features and segmentation encoded features are fused to obtain fused features, and the fused features and problem description information are processed by a large language model to obtain the processing result.

[0047] In the above embodiments, by performing downsampling and segmentation processing on the image to be processed, local and global information of the image to be processed can be extracted. By fusing the local and global information and then processing the fused features through a large language model, the fine-grained perception capability of the multimodal large model for high-resolution images can be improved, further enhancing the overall performance of the model. This can optimize the user experience and meet the needs of downstream tasks requiring fine-grained perception.

[0048] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0050] Figure 1 A flowchart of an image processing method based on a multimodal large model provided in an embodiment of this disclosure is shown;

[0051] Figure 2 This diagram illustrates a structural schematic of a multimodal large model provided in an embodiment of the present disclosure;

[0052] Figure 3 A schematic diagram of the processing flow of a dual-path visual input module in a multimodal large model provided in this disclosure embodiment is shown;

[0053] Figure 4A schematic diagram of the processing flow of a dual-path feature fusion module in a multimodal large model provided in this disclosure embodiment is shown;

[0054] Figure 5 A schematic diagram of an image processing apparatus based on a multimodal large model provided in an embodiment of this disclosure is shown.

[0055] Figure 6 A schematic diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0057] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0058] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0059] With the rise of generative large language models, text-generative multimodal large models have gradually come into view. Compared with traditional discriminative multimodal large models, text-generative multimodal large models, represented by LLAVA, have rapidly gained widespread recognition due to their convenient interaction, universality of capabilities, and good scalability.

[0060] In the conventional multimodal large models mentioned above, without increasing the resolution, most multimodal large models have very poor perception ability for large images. This directly leads to severe model illusions when there is a lot of text or small objects in the image, making it impossible to give reliable results.

[0061] To address this problem, those skilled in the art have proposed the following solutions:

[0062] Option 1: Train a visual transformer with a higher input resolution. However, this option requires a very large training overhead, and the larger resolution images require more storage space, which increases the training difficulty of this technique, making related experiments unlikely to yield results in the short term.

[0063] Option 2: This option uses image slicing to divide the large image into multiple resolution segments, then merges the results. This option doesn't require completely retraining the visual Transformer, resulting in lower training overhead compared to Option 1. However, because image slicing requires multiple inference iterations, its inference cost is higher. Furthermore, this option often requires the model to be trained from the beginning using large-resolution images.

[0064] Option 3: Use other detail-aware modules as aids, such as adding grounding and OCR models to the model. This option can better improve the performance of downstream tasks, but its versatility is poor and it is only suitable for some downstream tasks.

[0065] Based on the above research, this disclosure provides an image processing method, apparatus, device, and medium based on a multimodal large model. In the embodiments of this disclosure, firstly, an image to be processed and problem description information for the image to be processed are acquired; then, the image to be processed is subjected to downsampling and segmentation processing respectively to obtain downsampled feature maps from multiple feature extraction angles and multiple segmentation feature maps; next, the downsampled feature maps from multiple feature extraction angles and multiple segmentation feature maps are encoded by a visual encoder to obtain downsampled encoded features and segmentation encoded features; finally, the downsampled encoded features and segmentation encoded features are fused to obtain fused features, and the fused features and problem description information are processed by a large language model to obtain the processing result.

[0066] In the above embodiments, by performing downsampling and segmentation processing on the image to be processed, local and global information of the image to be processed can be extracted. By fusing the local and global information and then processing the fused features through a large language model, the fine-grained perception capability of the multimodal large model for high-resolution images can be improved, further enhancing the overall performance of the model. This can optimize the user experience and meet the needs of downstream tasks requiring fine-grained perception.

[0067] To facilitate understanding of this embodiment, a detailed description of the image processing method based on a multimodal large model disclosed in this disclosure will be provided first. The execution entity of the image processing method based on a multimodal large model provided in this disclosure is generally an electronic device with a certain computing power. In some possible implementations, this image processing method based on a multimodal large model can be implemented by a processor calling computer-readable instructions stored in memory.

[0068] See Figure 1 The diagram shows a flowchart of an image processing method based on a multimodal large model provided in this disclosure. The method includes steps S101 to S104, wherein:

[0069] S101: Obtain the image to be processed and the problem description information for the image to be processed.

[0070] Here, the problem description information is the question posed to the image to be processed. For example, the problem description information could be "What is in this image?"

[0071] S102: Perform downsampling and segmentation processing on the image to be processed to obtain downsampled feature maps and multiple segmentation feature maps with multiple feature extraction angles.

[0072] In this embodiment of the disclosure, the image to be processed can be downsampled from multiple feature extraction angles to obtain multiple downsampled feature maps from multiple feature extraction angles. For example, as Figure 3 As shown, for a single image to be processed, four downsampled feature maps can be obtained. Furthermore, the image can be segmented to obtain multiple sub-images, each of which is a segmentation feature map. For example, as... Figure 3 As shown, the image to be processed can be divided into 4 segmentation feature maps.

[0073] Here, by downsampling the image to be processed, global information of the image can be extracted from multiple feature extraction perspectives. By segmenting the image to be processed, local region information can be extracted.

[0074] S103: The downsampled feature map of the multi-feature extraction angle and the multiple segmentation feature maps are encoded by a visual encoder to obtain downsampled encoded features and segmentation encoded features.

[0075] In this embodiment of the disclosure, the downsampled feature map and the segmented feature map can be encoded by a visual encoder (Vision Transformer, or ViT for short) to obtain the downsampled encoded features and the segmented encoded features, respectively.

[0076] Here, the fused features of the downsampled feature map and the segmented feature map can be encoded by a visual encoder to obtain the downsampled encoded features and the segmented encoded features.

[0077] In addition, the fused features of the downsampled feature map and the segmented feature map can be encoded separately by a visual encoder to obtain the downsampled encoded features and the segmented encoded features in sequence.

[0078] S104: The downsampling coding features and the segmentation coding features are fused to obtain fused features, and the fused features and the problem description information are processed by a large language model to obtain the processing result.

[0079] In this embodiment of the disclosure, firstly, feature transformation can be performed on the downsampled coding features and the segmented coding features, and then the feature-transformed downsampled coding features and segmented coding features can be fused to obtain fused features. For example, feature transformation can be performed on the downsampled coding features from a feature dimension, and feature transformation can be performed on the segmented coding features from a spatial dimension.

[0080] In this embodiment of the disclosure, by fusing local and global information and processing the fused features through a large language model, the fine-grained perception capability of the multimodal large model for high-resolution images can be improved, further enhancing the overall performance of the model. This can optimize the user experience and meet the needs of downstream tasks requiring fine-grained perception.

[0081] The above steps will be explained in detail below with reference to specific implementation methods.

[0082] In this disclosure, the multimodal large model mainly refers to a generative visual text cross-modal large model that combines a visual encoder structure and a generative large language model structure. The specific visual encoder structure and generative large language model structure are not limited in this disclosure.

[0083] like Figure 2As shown, the multimodal large model in this disclosure specifically includes the following modules connected in sequence: a dual-path visual input module, a visual encoder, a dual-path feature fusion module, a mapping layer, and a decoder-only large language model structure. In this multimodal large model, the visual encoder is mainly a Visual Transformer (ViT) trained using a clip pre-training scheme or a similar scheme, with a basic resolution mostly between 224x224 pixels, 336x336 pixels, or 448x448 pixels. This disclosure does not limit the specific resolution and visual encoder structure; the specific implementation is the determining factor.

[0084] Based on this, the above steps perform downsampling and segmentation processing on the image to be processed, respectively, to obtain downsampled feature maps with multiple feature extraction angles and multiple segmentation feature maps, specifically including:

[0085] The image to be processed is downsampled and segmented by a dual-path visual input module to obtain downsampled feature maps and multiple segmented feature maps from multiple feature extraction angles.

[0086] The above steps fuse the downsampling coding features and the segmentation coding features to obtain fused features, specifically including:

[0087] The downsampled coding features and the segmented coding features are fused by a dual-path feature fusion module to obtain the fused features.

[0088] like Figure 2 As shown, firstly, the image to be processed is downsampled and segmented through a dual-path visual input module to obtain downsampled feature maps and multiple segmented feature maps. Then, a visual encoder encodes the downsampled feature maps and multiple segmented feature maps from multiple feature extraction angles to obtain downsampled encoded features and segmented encoded features. Next, a dual-path feature fusion module fuses the downsampled encoded features and segmented encoded features to obtain fused features. These fused features are then input into a large language model for processing after passing through a mapping layer. Simultaneously, the problem description information needs to be vectorized into text, and the resulting text vectors are input into the large language model for further processing to obtain the final result.

[0089] Here, the dual-path visual input module includes two processing modules: a visual downsampling module and a visual segmentation module; these modules can be executed in parallel. The visual downsampling module processes the image to be processed to obtain a downsampled feature map, while the visual segmentation module processes the image to be processed to obtain a segmented feature map.

[0090] Here, the dual-path feature fusion module includes two processing modules and a mapping and merging module, namely a first-path feature conversion module and a second-path feature conversion module; wherein, the first-path feature conversion module is used to perform feature conversion on the downsampled coded features, the second-path feature conversion module is used to perform feature conversion on the segmented coded features, and the mapping and merging module is used to fuse the feature-converted downsampled coded features and the segmented coded features to obtain fused features.

[0091] In the above embodiments, by setting up a dual-path visual input module and a dual-path feature fusion module, it is possible to extract local and global information of the image to be processed, and to fuse local and global information. By processing the fused features through a large language model, the fine-grained perception capability of the multimodal large model for high-resolution images can be improved, further enhancing the overall effect of the model. This can optimize the user experience and meet the needs of downstream tasks that require fine-grained perception.

[0092] In an optional implementation, step S102 above performs downsampling processing on the image to be processed to obtain downsampled feature maps with multiple feature extraction angles, specifically including the following steps:

[0093] Step S11: Perform downsampling processing on the image to be processed to obtain a downsampled image with multiple feature extraction angles;

[0094] Step S12: Normalize the size of the downsampled image for each of the multi-feature extraction angles;

[0095] Step S13: Determine the downsampled feature map of the multi-feature extraction angle based on the downsampled image after normalization.

[0096] For ease of description, this disclosed technical solution assumes that the basic resolution of ViT is S×S (S is an even number, and the unit is pixels), the number of channels is C (usually 3), the feature length of the last layer output is P×P+1, the dimension is D_v, and the image resolution scaling factor is k (k is an integer that must be divisible by S). The input batch_size is denoted as B.

[0097] Here, the visual downsampling module can be set as a k×k convolutional layer, where pad is 0, stride is k, input dimension is C, and output dimension is k. 2 C, the bias parameter is k 2 C. This convolutional layer can be followed by a layer normalization layer (LayerNorm) to limit the numerical range.

[0098] In practice, convolutional layers can be used to perform convolution calculations on the image to be processed, thereby obtaining multiple downsampled images; then, regularization layers are used to normalize the multiple downsampled images, thereby obtaining downsampled feature maps with multiple feature extraction angles.

[0099] For example, suppose the size of the dual-path visual input is (B, C, kS, kS). After processing by the above convolutional and regularization layers, we can obtain an input with size (B, kS). 2 The downsampled image after normalization of C,S,S).

[0100] In this embodiment of the disclosure, the above steps, based on the normalized downsampled image, determine the downsampled feature map of the multi-feature extraction angle, specifically including the following steps:

[0101] First, the multi-feature extraction angle downsampled image is segmented into feature maps according to the number of feature channels, resulting in multiple feature map blocks; wherein, the number of the multiple feature map blocks is the same as the number of multi-feature extraction angles;

[0102] Next, the multiple feature map blocks are stitched together to obtain the downsampled feature map.

[0103] After obtaining the size (B, k) 2 After normalizing the downsampled image (C, S, S), the second dimension of this feature (i.e., the dimension of the number of feature channels) can be divided into k equal parts according to C. 2 1. Feature map patches; finally, these feature map patches are stacked together in the first dimension (B), so that the output feature dimension is (k). 2 B, C, S, S). Therefore, the size of the downsampled feature map is (k 2 B,C,S,S).

[0104] The above implementation method can extract global information from the image, thereby providing feature basis for improving the overall effect of multimodal large models.

[0105] In an optional implementation, step S102 above performs segmentation processing on the image to be processed to obtain multiple segmentation feature maps, specifically including the following steps:

[0106] Step S21: Divide the image to be processed into multiple sub-images; wherein the number of the multiple sub-images is the same as the number of the multi-feature extraction angles;

[0107] Step S22: The multiple sub-images are stitched together to obtain the multiple segmentation feature maps.

[0108] In this embodiment of the disclosure, the image to be processed can be divided into multiple sub-images by a visual segmentation module, and then the multiple sub-images can be stitched together to obtain the multiple segmentation feature maps.

[0109] For an input of size (B, C, kS, kS), the visual segmentation module first segments the input into k×k small blocks (i.e., multiple sub-images) according to (B, C, S, S). Then, the multiple small blocks are stacked in the first dimension, and the feature dimension of the segmentation feature map output by the visual segmentation module is (k... 2 B,C,S,S).

[0110] In an optional implementation, step S103 above encodes the downsampled feature map of the multi-feature extraction angle and the multiple segmented feature maps using a visual encoder to obtain downsampled encoded features and segmented encoded features, specifically including the following steps:

[0111] Step S31: The downsampled feature map from the multiple feature extraction angles and the multiple segmented feature maps are fused to obtain the fused features;

[0112] Step S32: Input the fused features into the visual encoder for processing to obtain output features; wherein, the output features include the downsampled coding features and the segmentation coding features.

[0113] As can be seen from the above description in the embodiments of this disclosure, the fused features of the downsampled feature map and the segmented feature map can be encoded by a visual encoder to obtain the downsampled encoded features and the segmented encoded features.

[0114] After obtaining the size (k) 2 The downsampled feature map of B,C,S,S) is obtained, and the size of (k) is obtained. 2 After segmenting the feature maps into B, C, S, S), the two feature paths can be stacked in the first dimension to obtain (2k) 2 The final features of B, C, S, S) are the fused features. At this point, the features of size (2k...) can be... 2 The fused features of B, C, S, S are fed into ViT for video encoding processing. The output data of the video encoder includes downsampling coding features and segmentation coding features.

[0115] For an image input of size (B, C, kS, kS), the size of the output data after passing through the dual-path visual input module and ViT should be (2k... 2 B, P×P+1, D_v), after removing the category token, the size is (2k 2 B,P×P,D_v).

[0116] In an optional implementation, step S104 above fuses the downsampling coding features and the segmentation coding features to obtain fused features, specifically including the following steps:

[0117] Step S41: Perform feature fusion on the downsampled encoded features from the perspective of multi-feature extraction to obtain the first feature to be fused;

[0118] Step S42: Spatial fusion is performed on the segmented coding features from a spatial perspective to obtain the second feature to be fused;

[0119] Step S43: Determine the fusion feature based on the first feature to be fused and the second feature to be fused.

[0120] In this embodiment of the disclosure, for a size of (2k 2 The output data of B,P×P,D_v can be divided into two parts of size (k) 2 The sub-features of B,P×P,D_v are the downsampling coding feature and the segmentation coding feature, respectively.

[0121] Here, feature transformation can be performed on the downsampled encoded features from the feature dimension. This feature transformation can transform the size (k) of the downsampled encoded features. 2 The downsampled encoded features of size (B,P×P,D_v) are converted into features of size (B,P×P,k) 2 The features of D_v), i.e., the first feature to be fused. Feature transformation can also be performed on the segmented encoded features from the spatial dimension. This feature transformation can transform features of size (k) 2 The segmentation coding features of (B, P×P, D_v) are transformed into (B, k) 2 The features of P×P,D_v), namely, the second feature to be fused. Then, the fusion feature can be determined based on the first and second features to be fused.

[0122] The above processing methods enable feature completion of downsampled encoded features from the feature dimension and fusion of segmented encoded features from the spatial dimension. The fused features obtained using this method contain richer and more comprehensive information, thereby improving the fine-grained perception capability of multimodal large models for high-resolution images and further enhancing the overall performance of the model.

[0123] In this embodiment of the disclosure, feature fusion is performed on the downsampled encoded features from the perspective of multi-feature extraction to obtain a first feature to be fused, specifically including the following steps:

[0124] First, the downsampled encoded features are segmented to obtain the first sub-encoded features of each feature extraction angle in each feature channel;

[0125] Secondly, the first sub-encoded features are concatenated from the perspective of feature dimensions in the downsampled encoded features to obtain the first feature to be fused.

[0126] In this embodiment of the disclosure, in order to make the size (k 2 The downsampled encoded features of size (B,P×P,D_v) are converted into features of size (B,P×P,k) 2 The features of D_v can be divided into k average dimensions of the first dimension of the downsampled encoded feature. 2 There are 10 feature blocks (i.e., the first sub-encoded features). These feature blocks are then stacked uniformly on the third dimension D_v, resulting in a structure of size (B, P × P, k). 2 The features of D_v (i.e., the first feature to be fused).

[0127] Next, we can use an input dimension of k. 2 A 2D fully connected layer with an output dimension of D processes the first feature to be fused, and the processing result of the fully connected layer is processed by an activation layer to obtain a feature of size (B, P×P, D_v), denoted as feature 1.

[0128] In this embodiment of the disclosure, the segmented coding features are spatially fused from a spatial perspective to obtain a second feature to be fused, specifically including the following steps:

[0129] First, the segmented coding features are divided to obtain second sub-coding features for each spatial angle;

[0130] Secondly, the second sub-encoding features are spliced ​​together from the perspective of spatial dimension in the segmented encoding features to obtain the second feature to be fused.

[0131] In this embodiment of the disclosure, in order to make the size (k 2 The segmentation and coding features of B, P×P, D_v are transformed into (B, k) 2 The features of P×P,D_v) can first be divided into k equal parts based on the first dimension of the segmentation encoding feature. 2 There are 10 feature blocks (i.e., the second sub-encoded features). These feature blocks are then stacked uniformly on the second dimension P×P, resulting in a size (B, k). 2 The features of P×P,D_v (i.e., the second feature to be fused). Then, through an input dimension of k 2 The P×P output dimension is processed by a fully connected layer, and the processing result of the fully connected layer is processed by an activation layer to obtain a feature of size (B, P×P, D_v), which is denoted as feature 2.

[0132] Then, the two features (i.e., feature 1 and feature 2) can be concatenated in the last dimension to obtain a feature of (B, P×P, 2D_v). This feature is then processed by a fully connected layer with an input dimension of 2D_v and an output dimension of D_v. The output of this fully connected layer is then processed by an activation layer to transform the feature into a fused feature of (B, P×P, D_v).

[0133] As described above, the fused features incorporate both local and global information. These fused features are then processed into features with the same embedding feature dimension D_l as the text input, and a length of M. They are then inserted into the corresponding positions of the text features determined according to rules. All features are then fed into a multimodal large language model with only a decoder for inference.

[0134] In this embodiment of the disclosure, the multimodal large model can also be trained in the following ways:

[0135] First, training samples are obtained, and the dual-path visual input module and dual-path feature fusion module are trained using the training samples to obtain the trained dual-path visual input module and trained dual-path feature fusion module.

[0136] Secondly, the trained dual-path visual input module, the trained dual-path feature fusion module, the visual encoder, and the large language model are trained using the training samples to obtain the multimodal large model.

[0137] In this embodiment, the weights of a pre-trained multimodal large model are loaded as initial weights, and the weights of the dual-path visual input module and the dual-path feature merging module are randomly initialized. Then, the multimodal large model undergoes two-stage training on a high-resolution image-text dialogue dataset, specifically including the following:

[0138] In the first training phase, the visual encoder, mapping layer module, and large language model module are not involved in the training process. The dual-path visual input module and dual-path feature merging module are trained until their training accuracy meets the training requirements. The main purpose of model training in this phase is to align the model with the training task and construct reasonable initialization parameters for the added parameters.

[0139] In the second training phase, all parameters participate in the training. This involves training the trained dual-path visual input module, the trained dual-path feature fusion module, the visual encoder, and the large language model using training samples until these modules meet the training requirements, resulting in a multimodal large model. The main purpose of model training in this phase is to jointly learn the ability to expand resolution across all modules.

[0140] This disclosed technical solution does not limit the specific training hyperparameters. It also does not limit whether an efficient parameter fine-tuning scheme (such as LoRA) is used in the training of large language model structures.

[0141] In this embodiment, the amount of fused features obtained through the method described above is smaller, thereby reducing the data processing load of the multimodal model, making model training easier and more cost-effective. The dual-path visual input module and dual-path feature fusion module added in this technical solution have fewer parameters, thus making the training process of the multimodal model easier.

[0142] The following is combined with Figures 2 to 4 The above process will be described.

[0143] like Figure 2 As shown, the multimodal large model specifically includes the following modules connected in sequence: a dual-path visual input module, a visual encoder, a dual-path feature fusion module, a mapping layer, and a decoder-only large language model structure.

[0144] Assume the image to be processed is a picture, the text length of the problem description information is T, the position of inserting visual features (i.e., fusion features) is m, the length of the visual features is M, and the embedding dimension of the large language model structure for the text input is D_l.

[0145] First, the image is processed by a preprocessing module into an array of size (1, C, kS, kS). Then, it is input into a dual-path visual input module for processing, resulting in an array of size (2k... 2 The input features of C, S, S) (i.e., downsampled feature map and segmented feature map).

[0146] like Figure 3 As shown, the array (1, C, kS, kS) is first processed by the visual downsampling module to obtain an array of size (kS). 2 The downsampled feature map of (1, C, S, S) is obtained; at the same time, the array of (1, C, kS, kS) needs to be processed by the visual segmentation module to obtain the size of (k 2 The two feature maps (C, S, S) are segmented. These two feature maps are stacked in the first dimension to obtain a result of size (2k). 2 Input features of C,S,S).

[0147] Next, the visual input module is used to input data of size (2k) 2 The input features (C,S,S) are visually encoded to obtain a result of size (2k). 2 The coding features of P×P,D_v (i.e., downsampling coding features and segmentation coding features).

[0148] Then, a dual-path feature fusion module can be used to process features of size (2k). 2 The encoded features of (1, P×P, D_v) are processed to obtain two features of scale (1, P×P, 2D_v) (i.e., fused features).

[0149] like Figure 4 As shown, the encoded features (2k 2 The P×P,D_v) block is divided into two parts with a size of (k 2 The sub-features are (1, P×P, D_v). For one of the sub-features, after feature transformation, fully connected layer and activation layer, we can get feature 1 with size (1, P×P, D_v); for the other sub-feature, after feature transformation, fully connected layer and activation layer, we can get feature 2 with size (1, P×P, D_v); concatenating the two features in the last dimension, we get the fused feature (B, P×P, 2D_v).

[0150] Next, the fused features can be processed into features with the same embedding feature dimension D_l as the text input, and a length of M, resulting in a (1,M,D_l) feature. Then, this feature is inserted into the m-th position of the encoded and embedded text vector (1,T,D_l), resulting in a feature with shape (1,T+M,D_l). Subsequently, this feature is fed into the decoder of the large language model structure, and the final processing result is output.

[0151] As described above, this disclosed technical solution designs two trainable modules: a dual-path visual input module and a dual-path feature merging module. A dedicated training scheme is designed for the newly added modules. The two newly added trainable modules in this disclosed technical solution have a small number of parameters, making training relatively easy. After processing the image using this disclosed technical solution, the feature shape of the input model is more fixed, allowing it to be made into a static image, thereby improving training and inference efficiency. Furthermore, this disclosed technical solution integrates local and global information in its design; therefore, the multimodal large model of this disclosed technical solution has superior fine-grained perception capability for high-resolution images and superior perception capability for low-resolution images, thus further improving the overall performance of the model.

[0152] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0153] Based on the same inventive concept, this disclosure also provides an image processing apparatus based on a multimodal large model, which corresponds to the image processing method based on a multimodal large model. Since the principle of the apparatus in this disclosure for solving the problem is similar to the image processing method based on a multimodal large model described above, the implementation of the apparatus can refer to the implementation of the method, and the repeated parts will not be described again.

[0154] Reference Figure 5 The diagram shown is a schematic of an image processing device based on a multimodal large model according to an embodiment of this disclosure. The device includes: an acquisition module 10, a dual-channel visual input module 20, a visual encoder 30, and a dual-channel feature fusion module 40; wherein,

[0155] The acquisition module is used to acquire the image to be processed and the problem description information for the image to be processed;

[0156] The dual-path visual input module is used to perform downsampling and segmentation processing on the image to be processed, respectively, to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles;

[0157] A visual encoder is used to encode the downsampled feature map of the multi-feature extraction angle and the multiple segmented feature maps to obtain downsampled encoded features and segmented encoded features.

[0158] The dual-path feature fusion module is used to fuse the downsampled coding features and the segmented coding features to obtain fused features, and then process the fused features and the problem description information through a large language model to obtain the processing result.

[0159] In one possible implementation,

[0160] The dual-path visual input module is further configured to: perform downsampling and segmentation processing on the image to be processed through the dual-path visual input module to obtain downsampled feature maps and multiple segmentation feature maps with multiple feature extraction angles; the dual-path feature fusion module is further configured to: fuse the downsampled encoded features and the segmentation encoded features through the dual-path feature fusion module to obtain the fused features.

[0161] In one possible implementation, the dual-path visual input module is further configured to: perform downsampling processing on the image to be processed to obtain downsampled images with multiple feature extraction angles; perform size normalization processing on the downsampled images with each of the multiple feature extraction angles; and determine the downsampled feature map of the multiple feature extraction angles based on the downsampled images after normalization processing.

[0162] In one possible implementation, the dual-path visual input module is further configured to: segment the multi-feature extraction angle downsampled image into feature maps according to the number of feature channels of the multi-feature extraction angle downsampled image to obtain multiple feature map blocks; wherein the number of the multiple feature map blocks is the same as the number of multi-feature extraction angles; and stitch the multiple feature map blocks together to obtain the downsampled feature map.

[0163] In one possible implementation, the dual-path visual input module is further configured to: divide the image to be processed into multiple sub-images; wherein the number of the multiple sub-images is the same as the number of the multiple feature extraction angles; and stitch the multiple sub-images together to obtain the multiple segmentation feature maps.

[0164] In one possible implementation, the visual encoder is further configured to: fuse the downsampled feature map of the multi-feature extraction angle and the multiple segmented feature maps to obtain fused features; input the fused features into the visual encoder for processing to obtain output features; wherein the output features include the downsampled encoded features and the segmented encoded features.

[0165] In one possible implementation, the dual-path feature fusion module is further configured to: perform feature fusion on the downsampled encoded features from the perspective of multi-feature extraction to obtain a first feature to be fused; perform spatial fusion on the segmented encoded features from a spatial perspective to obtain a second feature to be fused; and determine the fused feature based on the first feature to be fused and the second feature to be fused.

[0166] In one possible implementation, the dual-path feature fusion module is further configured to: segment the downsampled coded features to obtain first sub-coded features with each feature extraction angle in each feature channel; and concatenate the first sub-coded features from the perspective of the feature dimension in the downsampled coded features to obtain the first feature to be fused.

[0167] In one possible implementation, the dual-path feature fusion module is further configured to: segment the segmented coding features to obtain second sub-coding features from each spatial angle; and concatenate the second sub-coding features from the spatial dimension of the segmented coding features to obtain the second feature to be fused.

[0168] In one possible implementation, the device is further configured to: acquire training samples, and train the dual-path visual input module and the dual-path feature fusion module using the training samples to obtain the trained dual-path visual input module and the trained dual-path feature fusion module; train the trained dual-path visual input module, the trained dual-path feature fusion module, the visual encoder, and the large language model using the training samples to obtain the multimodal large model after training.

[0169] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0170] Corresponding to Figure 1 The present disclosure also provides an electronic device 600, such as an image processing method based on a multimodal large model. Figure 6 The diagram shown is a structural schematic of an electronic device 600 provided in an embodiment of this disclosure, including:

[0171] The system includes a processor 61, a memory 62, and a bus 63. The memory 62 stores execution instructions and includes main memory 621 and external memory 622. The main memory 621, also called internal memory, temporarily stores the computational data in the processor 61, as well as data exchanged with external memory such as a hard disk. The processor 61 exchanges data with the external memory 622 through the main memory 621. When the electronic device 600 is running, the processor 61 communicates with the memory 62 through the bus 63, causing the processor 61 to execute the following instructions:

[0172] Obtain the image to be processed and the problem description information for the image to be processed;

[0173] The image to be processed is subjected to downsampling and segmentation processing respectively to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles;

[0174] The downsampled feature map of the multi-feature extraction angle and the multiple segmented feature maps are encoded by a visual encoder to obtain downsampled encoded features and segmented encoded features.

[0175] The downsampling coding features and the segmentation coding features are fused to obtain fused features. The fused features and the problem description information are then processed using a large language model to obtain the processing result.

[0176] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the image processing method based on a multimodal large model described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0177] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the image processing method based on a multimodal large model described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0178] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0182] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. An image processing method based on a multimodal large model, characterized in that, include: Obtain the image to be processed and the problem description information for the image to be processed; The image to be processed is subjected to downsampling and segmentation processing respectively to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles; The downsampled feature map of the multi-feature extraction angle and the multiple segmented feature maps are encoded by a visual encoder to obtain downsampled encoded features and segmented encoded features. The downsampling coding features and the segmentation coding features are fused to obtain fused features. The fused features and the problem description information are then processed using a large language model to obtain the processing result.

2. The method according to claim 1, characterized in that, The step of performing downsampling and segmentation processing on the image to be processed to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles includes: performing downsampling and segmentation processing on the image to be processed through a dual-path visual input module to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles. The step of fusing the downsampled coding features and the segmented coding features to obtain fused features includes: fusing the downsampled coding features and the segmented coding features through a dual-path feature fusion module to obtain the fused features.

3. The method according to claim 1 or 2, characterized in that, The step of downsampling the image to be processed to obtain downsampled feature maps from multiple feature extraction angles includes: The image to be processed is downsampled to obtain a downsampled image with multiple feature extraction angles; The size of the downsampled image for each of the multi-feature extraction angles is normalized. Based on the downsampled image after normalization, the downsampled feature map of the multi-feature extraction angle is determined.

4. The method according to claim 3, characterized in that, The step of determining the downsampled feature map of the multi-feature extraction angle based on the normalized downsampled image includes: Based on the number of feature channels in the downsampled image from the multi-feature extraction angle, the downsampled image from the multi-feature extraction angle is segmented into feature maps to obtain multiple feature map patches; wherein, the number of the multiple feature map patches is the same as the number of multi-feature extraction angles; The multiple feature map blocks are concatenated to obtain the downsampled feature map.

5. The method according to claim 1 or 2, characterized in that, The image to be processed is segmented to obtain multiple segmentation feature maps, including: The image to be processed is divided into multiple sub-images; wherein the number of the multiple sub-images is the same as the number of the multi-feature extraction angles; The multiple sub-images are stitched together to obtain the multiple segmentation feature maps.

6. The method according to claim 1, characterized in that, The step involves encoding the downsampled feature maps from the multiple feature extraction angles and the multiple segmented feature maps using a visual encoder to obtain downsampled encoded features and segmented encoded features, including: The downsampled feature map from the multiple feature extraction angles and the multiple segmented feature maps are fused to obtain the fused features; The fused features are input into the visual encoder for processing to obtain output features; wherein, the output features include the downsampled coding features and the segmented coding features.

7. The method according to claim 1 or 2, characterized in that, The process of fusing the downsampled coding features and the segmented coding features to obtain fused features includes: The downsampled encoded features are fused from the perspective of multi-feature extraction to obtain the first feature to be fused. The segmented coding features are spatially fused from a spatial perspective to obtain a second feature to be fused. The fusion feature is determined based on the first feature to be fused and the second feature to be fused.

8. The method according to claim 7, characterized in that, The step of fusing the downsampled encoded features from the perspective of multi-feature extraction to obtain the first feature to be fused includes: The downsampled encoded features are segmented to obtain the first sub-encoded features of each feature extraction angle in each feature channel; The first sub-coded features are concatenated from the feature dimension of the downsampled encoded features to obtain the first feature to be fused.

9. The method according to claim 7, characterized in that, The spatial fusion of the segmented coding features from a spatial perspective to obtain the second feature to be fused includes: The segmented coding features are divided to obtain second sub-coding features for each spatial angle; The second sub-encoded feature is concatenated from the spatial dimension of the segmented encoded feature to obtain the second feature to be fused.

10. The method according to claim 2, characterized in that, The method also includes training a multimodal large model in the following ways: Acquire training samples, and train the dual-path visual input module and the dual-path feature fusion module using the training samples to obtain the trained dual-path visual input module and the trained dual-path feature fusion module; The training samples are used to train the trained dual-path visual input module, the trained dual-path feature fusion module, the visual encoder, and the large language model to obtain the multimodal large model.

11. An image processing device based on a multimodal large model, characterized in that, include: The acquisition module is used to acquire the image to be processed and the problem description information for the image to be processed; The dual-path visual input module is used to perform downsampling and segmentation processing on the image to be processed, respectively, to obtain downsampled feature maps and multiple segmentation feature maps from multiple feature extraction angles; A visual encoder is used to encode the downsampled feature map of the multi-feature extraction angle and the multiple segmented feature maps to obtain downsampled encoded features and segmented encoded features. The dual-path feature fusion module is used to fuse the downsampled coding features and the segmented coding features to obtain fused features, and then process the fused features and the problem description information through a large language model to obtain the processing result.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the image processing method based on a multimodal large model as described in any one of claims 1 to 9.

13. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the image processing method based on a multimodal large model as described in any one of claims 1 to 9.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the image processing method based on a multimodal large model as described in any one of claims 1 to 9.

Images