Multi-modal understanding optimization method based on fine-grained feature extraction and global information integration

Abstract

The invention relates to a multi-modal understanding optimization method based on fine-grained feature extraction and global information integration. The method comprises the following steps: segmenting an input image into a plurality of local image blocks, extracting local fine-grained visual features and global visual features, and interacting the local fine-grained visual features and the global visual features based on an attention mechanism to obtain local context features; secondly, performing feature fusion on the local fine-grained visual features to generate fused visual features; mapping the fused visual features to a semantic space which is the same as the text features of the large language model to obtain projected visual features, and dynamically screening through attention weight based on the text features to obtain key visual features; and fusing the key visual features with the text features to generate joint feature representation, and inputting the joint feature representation into a large language model to generate a semantic analysis result. According to the method, global information and dynamic feature selection are introduced, so that the ability of the model to understand multi-modal content in a complex scene can be improved, and the model calculation overhead is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and particularly relates to a multi-modal understanding optimization method based on fine-grained feature extraction and global information integration. Background Art

[0002] With the rapid development of technologies in the field of artificial intelligence, multi-modal content understanding technologies based on deep learning have emerged. Multi-modal content understanding technologies can comprehensively understand the information of each modal content by processing data of multiple modalities such as images, texts, videos, and audios. And with the in-depth research, multi-modal large language models have emerged. Multi-modal large language models can integrate visual encoders into large language models and utilize the "prior knowledge" about the world mastered by large language models after a large amount of training to enhance the performance of multi-modal models and reduce the computational overhead of traditional multi-modal models.

[0003] In traditional technologies, to improve the understanding ability of multi-modal large language models for image details, high-resolution images are usually used for training, and a cropping strategy is adopted before input to improve training efficiency. However, the current traditional methods relying on high-resolution images and cropping strategies have many problems. On the one hand, the cropping strategy often causes the objects in the image to be separated from their surrounding connection areas, ignoring the overall context information of the image, which is particularly obvious in small objects or objects with irregular shapes. For example, in an optical character recognition task (OCR), the word "Breakdown" in a picture may be divided into "Break" and "down", causing semantic damage and affecting the understanding accuracy of downstream tasks. On the other hand, due to the fragmentation of local features, it is difficult for the model to accurately capture the semantic relationship between an object and its environment, resulting in poor performance when dealing with tasks that require fine classification and understanding. In addition, the input of high-resolution images and the use of image processing modules also lead to significant computational overhead, limiting the real-time performance and scalability of the model and making it difficult to deploy the model on resource-constrained devices. Summary of the Invention

[0004] Based on this, it is necessary to provide a multi-modal understanding optimization method based on fine-grained feature extraction and global information integration for the above technical problems to further improve the accuracy of multi-modal content understanding.

[0005] In a first aspect, the present application provides a multi-modal understanding optimization method based on fine-grained feature extraction and global information integration, including:

[0006] Dividing an input image into multiple local image patches, and based on a pre-trained model, extracting local fine-grained visual features of each local image patch and global visual features of the input image;

[0007] Based on the attention mechanism, the local fine-grained visual features are interacted with the global visual features to obtain local context features;

[0008] The local fine-grained visual features and the corresponding local context features are fused to generate fused visual features;

[0009] The fused visual features are mapped to the same semantic space as the text features of the large language model to obtain the projected visual features, and based on the input text features, the projected visual features are dynamically screened through attention weights to obtain key visual features;

[0010] The key visual features and the text features are fused to generate a joint feature representation, and the joint feature representation is input into the large language model to generate a semantic parsing result.

[0011] In one embodiment, based on the attention mechanism, the local fine-grained visual features are interacted with the global visual features to obtain local context features, including:

[0012] Taking the local fine-grained visual features as query vectors, taking the global visual features as key vectors and value vectors, and calculating attention scores;

[0013] Normalize the attention scores to generate the normalized first attention weights;

[0014] Fuse the normalized first attention weights with the global visual features to obtain local context features.

[0015] In one embodiment, based on the input text features, the projected visual features are dynamically screened through attention weights to obtain key visual features, including:

[0016] Concatenate the projected visual features with the global visual features to obtain concatenated features;

[0017] Add positional encodings to the concatenated features and the text features respectively to obtain the encoded concatenated features and the encoded text features;

[0018] Based on the attention mechanism of the large language model, calculate the second attention weights between the encoded concatenated features and the encoded text features to generate an attention weight map;

[0019] According to a preset threshold or weight sorting rule, dynamically select the visual features with second attention weights higher than the preset threshold based on the attention weight map to obtain key visual features.

[0020] In one embodiment, the local context features are calculated by the following formula:

[0021]

[0022] c i = softmax(attn) · g

[0023] where attn is the attention score, g is the global visual feature, and d i is the visual feature representation of the i-th local image patch, i.e., the local fine-grained visual feature, and d k is the dimension of the feature vector, and c i is the local context feature of the i-th local image patch.

[0024] In one embodiment, the second attention weight is calculated by the following formula:

[0025]

[0026] where attn' is the second attention weight, PE(*) is the position encoding function, t is the text feature, V' is the projected visual feature, g is the global visual feature, Concat(*) is used to concatenate two or more vectors within the parentheses, and d k is the dimension of the feature vector.

[0027] In one embodiment, the local fine-grained visual feature and the global visual feature are interacted based on the attention mechanism to obtain the local context feature, including:

[0028] Based on the local fine-grained visual feature and the global visual feature, calculate the initial attention scores of each local fine-grained visual feature and the global visual feature;

[0029] Perform sparsification processing on the initial attention scores corresponding to each local fine-grained visual feature, and filter out the set of region indices corresponding to the Top-K scores. The Top-K scores are the top K initial attention scores when the initial attention scores are sorted from high to low. Here, K is a preset sparsity parameter;

[0030] Perform Softmax normalization on the Top-K scores to generate the association weights between the local fine-grained visual feature and the global visual feature;

[0031] Based on the association weights, perform weighted summation on the global visual feature to generate the local context feature.

[0032] 7. According to the method of claim 1, after interacting the local fine-grained visual feature and the global visual feature based on the attention mechanism to obtain the local context feature, it further includes:

[0033] Centered on each local fine-grained visual feature, extract the adjacent local fine-grained visual features within the preset sliding window through the preset sliding window to generate a local context candidate set;

[0034] Based on the multi-head attention mechanism, calculate the association scores between each local fine-grained visual feature and the corresponding local context candidate set;

[0035] Add the association scores to the corresponding local fine-grained visual features to generate local enhanced features, and use the local enhanced features as the updated representation of the local context features.

[0036] In a second aspect, the present application also provides a multi-modal understanding optimization system based on fine-grained feature extraction and global information integration. The system includes:

[0037] An image feature extraction module, configured to divide an input image into multiple local image patches, and based on a pre-trained model, extract the local fine-grained visual features of each local image patch and the global visual feature of the input image;

[0038] A global feature fusion module, configured to:

[0039] Based on the attention mechanism, interact the local fine-grained visual features with the global visual features to obtain local context features;

[0040] Fuse the local fine-grained visual features and the corresponding local context features to generate fused visual features;

[0041] A dynamic feature selection module, configured to map the fused visual features to the same semantic space as the text features of the large language model to obtain projected visual features, and based on the input text features, dynamically screen the projected visual features through attention weights to obtain key visual features;

[0042] An image semantic parsing module, configured to fuse the key visual features and the text features to generate a joint feature representation, and input the joint feature representation into the large language model to generate a semantic parsing result.

[0043] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in the first aspect are implemented.

[0044] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the first aspect are implemented.

[0045] The above multi-modal understanding optimization method based on fine-grained feature extraction and global information integration provides basic data for subsequent feature fusion and analysis by splitting the input image into multiple local image patches and extracting the local fine-grained visual features of the local image patches and the global visual features of the input image. Secondly, based on the attention mechanism, local context features are generated through the interaction between the local fine-grained visual features and the global visual features, enabling the subsequent model to fully capture the correlation information between the local features and the global background and enhancing the understanding of the image content. Moreover, this method generates fused visual features by fusing the local fine-grained visual features and the corresponding local context features, enhancing the depth of understanding of the subsequent model for the context semantic environment, effectively solving the problem that existing multi-modal content understanding technologies are difficult to integrate context information when dealing with fine-grained targets, and improving the accuracy of overall content understanding. The obtained fused visual features are mapped to the same semantic space as the text features of the large language model to obtain the projected visual features, and combined with the text features, key visual features are dynamically selected through attention weights, which not only effectively reduces redundant information but also reduces the computational amount of the subsequent model and improves the feature processing efficiency. Finally, the key visual features and the text features are fused to generate a joint feature representation, which is input into the large language model to generate a semantic parsing result, realizing the accurate understanding and output of multi-modal content.

[0046] Compared with traditional multi-modal content understanding methods, this method optimizes the multi-modal content understanding process by introducing global features and dynamic feature selection, improves the model's understanding ability for complex scenarios and fine-grained content, and effectively reduces the computational cost while ensuring the model performance, enhancing the real-time performance and scalability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0048] Figure 1 Flowchart of the multi-modal understanding optimization method based on fine-grained feature extraction and global information integration provided by an exemplary embodiment of the present invention;

[0049] Figure 2 Schematic structural diagram of the multi-modal understanding optimization system based on fine-grained feature extraction and global information integration provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] In order to make the objectives, technical solutions, and advantages of this application clearer and more understandable, the following further details this application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0051] In one embodiment, as Figure 1 shown, a multi-modal understanding optimization method based on fine-grained feature extraction and global information integration is provided. In this embodiment, an example is given where this method is applied to a terminal. It can be understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0052] S101: Split the input image into multiple local image patches, and based on a pre-trained model, extract the local fine-grained visual features of each local image patch and the global visual feature of the input image.

[0053] Specifically, by splitting the input image into multiple local image patches, each local area of the image can be analyzed more carefully, and rich local detail information can be extracted. Schematically, this splitting method can be carried out according to a preset rule, for example, evenly divided into several rectangular patches of the same size, or can be divided based on the semantic information of the image content to ensure that each local image patch can contain an image area with certain semantic significance. Secondly, a pre-trained model such as a convolutional neural network model can be used to extract features from the local image patches and the complete input image. Among them, the local fine-grained visual features can reflect the feature information at the microscopic level such as texture, edge, and color distribution in the image, which is helpful for accurately identifying the local content of the image. The global visual feature can represent the information at the macroscopic level such as the overall layout of the image, the relative positions of the main objects, and the overall tone, providing a basis for understanding the overall semantics of the image.

[0054] S102: Based on the attention mechanism, interact the local fine-grained visual features with the global visual features to obtain local context features.

[0055] Specifically, after obtaining the local fine-grained visual features and the global visual features, an attention mechanism can be introduced to make them interact with each other, thereby obtaining local context features. Among them, through the attention mechanism, the human visual system can be simulated when observing an image, automatically focusing on certain key regions, and understanding local features based on global information. That is, through the attention mechanism, in the interaction process between the local fine-grained visual features and the global visual features, the local fine-grained visual features can be adjusted and weighted according to the context information provided by the global features. Schematically, if the global features indicate the existence of a certain specific object in the image, the local fine-grained visual features related to the object will be assigned a higher weight during the interaction process. On the contrary, the weights of the local features of non-key regions or irrelevant objects will be relatively reduced. The finally obtained local context features can better reflect the actual meaning and importance of the local image patch in the global background, enhancing the feature expression ability and semantic relevance.

[0056] S103: Feature fusion is performed on the local fine-grained visual features and the corresponding local context features to generate fused visual features.

[0057] Specifically, the local fine-grained visual features provide rich local detail information, while the local context features incorporate the weight adjustment information after global information semantic understanding. Based on the local fine-grained visual features and the local context features, a specific fusion algorithm or neural network structure can be designed to deeply fuse these two features in dimensions such as channels and space, so that the fused visual features not only retain the richness of local details but also can reflect the semantic role and relative importance of the local features in the entire image. Through this fusion process, the model can pay attention to both the details of the image and its surrounding context environment in subsequent processing, thereby more accurately understanding the image content.

[0058] S104: The fused visual features are mapped to the same semantic space as the text features of the large language model to obtain the projected visual features, and based on the input text features, the projected visual features are dynamically screened through attention weights to obtain the key visual features.

[0059] Specifically, a mapping function or neural network model can be constructed to map the fused visual features into the same semantic space as the text features of the large language model, obtaining the projected visual features. Through this mapping, the visual features can be converted into feature vectors with a semantic expression form similar to the text features, enabling them to interact and compare with the text features within the same semantic framework, thereby achieving cross-modal feature alignment. In addition, the text features can provide semantic descriptions and constraint information about the image content. The attention weights calculated based on the text features can reflect the relevance between each projected visual feature and the text semantics. Therefore, visual features with higher relevance are assigned larger weights, and key visual features are selected through the weight values. This process performs dynamic feature selection through the text features, which can extract the most matching and relevant parts from a large number of visual features according to the semantic requirements of the text, further focusing on the regions and details in the image that are closely related to the text semantics, and improving the pertinence and effectiveness of multi-modal fusion.

[0060] S105: Fuse the key visual features with the text features to generate a joint feature representation, and input the joint feature representation into the large language model to generate a semantic parsing result.

[0061] Specifically, the key visual features include important visual information in the image that is related to the text semantics, while the text features provide clear semantic descriptions and language logic. A fusion model or algorithm can be constructed to deeply fuse the two in the feature space to obtain a joint feature representation that can simultaneously reflect visual and semantic information. Finally, the joint feature representation can be input into the large language model for semantic parsing. Schematically, the large language model can perform operations such as semantic reasoning, logical analysis, and text generation based on the joint feature representation, and then generate an accurate and detailed semantic parsing result. This result can not only reflect the content description of the image but also be deeply understood in combination with the semantic logic of the text, thereby achieving efficient understanding and accurate semantic parsing of multi-modal fine-grained content, providing strong support for subsequent tasks such as image retrieval, content generation, and intelligent question answering.

[0062] In the above multi-modal understanding optimization method based on fine-grained feature extraction and global information integration, by splitting the input image into multiple local image patches and extracting local fine-grained visual features and global visual features, the local detailed information and global background information of the image can be effectively extracted, providing basic data for subsequent feature fusion. Secondly, by using the attention mechanism, the local fine-grained visual features and global visual features are interacted to obtain local context features, and the local context features are fused with the local fine-grained visual features to obtain fused visual features. This process enhances the subsequent model's ability to capture fine-grained target context information by introducing global information, effectively solving the problems of object and connection region segmentation and ignoring overall context information caused by traditional cropping strategies. Then, the projected visual features after mapping the fused visual features are combined with the text features, and key visual features are dynamically screened out through attention weights, effectively removing a large number of redundant visual features and reducing the computational overhead of the subsequent model. Finally, the key visual features and text features are fused to generate a joint feature representation, which is input into the large language model, and more accurate and comprehensive semantic parsing results can be obtained. This result can not only accurately reflect the multi-modal information contained in the image and text, but also deeply understand the semantic associations between modalities in complex scenarios, greatly improving the accuracy and practicality of multi-modal content understanding.

[0063] In one embodiment, based on the attention mechanism, the local fine-grained visual features and global visual features are interacted to obtain local context features, including:

[0064] Taking the local fine-grained visual features as query vectors, taking the global visual features as key vectors and value vectors, and calculating attention scores;

[0065] Normalizing the attention scores to generate the first normalized attention weights;

[0066] Fusing the first normalized attention weights with the global visual features to obtain local context features.

[0067] Specifically, the attention score can be obtained by calculating the similarity between the local fine-grained visual features and the global visual features. This attention score reflects the degree of correlation between each local fine-grained visual feature and different parts of the global visual features. Schematically, the softmax function can transform an arbitrary real-number vector input into a probability distribution vector, where each element in the vector is between 0 and 1, and the sum of all elements is 1. Then, the calculated attention score can be normalized by the softmax function to obtain the first attention weight, ensuring the rationality and comparability of the attention score. Subsequently, the normalized first attention weight can be dot-multiplied with the global visual features to obtain the local context features. Through this weighted fusion method, the local context features can better reflect the actual meaning and importance of the local image patches in the global background, enhancing the feature expression ability and semantic relevance.

[0068] In one embodiment, the local context features are calculated by the following formula:

[0069]

[0070] c i = softmax(attn)·g

[0071] where attn is the attention score, g is the global visual feature, d i is the visual feature representation of the i-th local image patch, that is, the local fine-grained visual feature. Dividing by is the dimension of the feature vector, and c i is the local context feature of the i-th local image patch.

[0072] Specifically, g T is the transpose of the global visual feature g, used for dot-multiplication with the local fine-grained visual feature d i Performing the dot product operation. Dividing the dot product result by Thereby avoiding problems of excessive numerical values or instability caused by a large dimension of the feature vector in subsequent calculations. After normalizing attn by the softmax function and performing a dot product operation with the global visual feature g, that is, each part of the global visual feature g is weighted and combined according to the corresponding attention weight to obtain the local context features.

[0073] In one embodiment, based on the input text features, the projected visual features are dynamically screened by attention weights to obtain the key visual features, including:

[0074] Concatenating the projected visual features and the global visual features to obtain the concatenated features;

[0075] Add positional encodings to the concatenated features and text features respectively to obtain the encoded concatenated features and encoded text features;

[0076] Based on the attention mechanism of the large language model, calculate the second attention weights between the encoded concatenated features and the encoded text features to generate an attention weight map;

[0077] According to a preset threshold or weight sorting rule, dynamically select visual features with second attention weights higher than the preset threshold based on the attention weight map to obtain key visual features.

[0078] Specifically, the fused visual features can be encoded and projected through a vision Transformer to generate projected visual features. These projected visual features contain local fine-grained features and their context information, which are suitable for capturing the details of the target object. The global visual features are directly extracted from the complete image, retaining the overall semantics of the image such as scene category, subject object distribution, etc. Therefore, in dynamic feature selection, introducing global visual features can provide higher-level semantic guidance and make decisions based on uncompressed global information, enhancing the robustness of the screening. Text features can provide semantic descriptions and constraint information about the image content. Concatenating the projected visual features and the global visual features in the feature dimension improves the accuracy of dynamic feature selection.

[0079] Positional encoding is usually used in the self-attention mechanism and can help the model distinguish features at different positions. Adding positional encoding to the concatenated features and text features can introduce the position information of the features, enabling the model to better understand the spatial and semantic relationships between the features. In addition, the attention weight map can be obtained by calculating the similarity between the encoded concatenated features and the encoded text features, that is, the second attention weights. Among them, the similarity can be calculated by dot product or other similarity measurement methods. This attention weight map quantifies the importance of visual features, provides a basis for subsequent dynamic screening, and reduces the computational complexity. And in the attention weight map, visual features higher than the preset threshold can be screened out as key visual features by the threshold method, or key visual features can be obtained by the sorting method using the weight sorting rule.

[0080] In one of the embodiments, the second attention weights are calculated by the following formula:

[0081]

[0082] where attn' is the second attention weight, PE(*) is the positional encoding function, t is the text feature, V' is the projected visual feature, g is the global visual feature, Concat(*) is used to concatenate two or more vectors within the parentheses, and d kis the dimension of the feature vector.

[0083] Specifically, Concat(V', g) is used to represent the concatenation of the projected visual feature V' and the global visual feature g. And this formula adds the encoded concatenated feature after position encoding and the encoded text feature to their corresponding unencoded original features respectively, and then performs matrix operations to measure the correlation degree between the text feature and the concatenated feature, obtaining a numerical set that can reflect their mutual relationship. Divide the result of this matrix operation by to perform normalization, avoid the influence of the feature vector dimension resulting in too large or unstable values, and ensure that the calculation result is within a reasonable range. And process the normalized result through the softmax function to transform it into a probability distribution form, obtaining the second attention weight attn'.

[0084] In one embodiment, based on the attention mechanism, the local fine-grained visual feature and the global visual feature are interacted to obtain the local context feature, including:

[0085] Based on the local fine-grained visual feature and the global visual feature, calculate the initial attention scores of each local fine-grained visual feature and the global visual feature;

[0086] Perform sparsification processing on the initial attention scores corresponding to each local fine-grained visual feature, and screen out the set of region indices corresponding to the Top-K scores. The Top-K scores are the top K initial attention scores when the initial attention scores are sorted from high to low, where K is a preset sparsity parameter;

[0087] Perform Softmax normalization on the Top-K scores to generate the correlation weight between the local fine-grained visual feature and the global visual feature;

[0088] Based on the correlation weight, perform weighted summation on the global visual feature to generate the local context feature.

[0089] Specifically, the initial attention scores can be obtained by calculating the similarity between the local fine-grained visual features and the global visual features, such as cosine similarity. Sort the initial attention scores from high to low, and select the top K initial attention scores as the Top-K scores. The set of region indices corresponding to the Top-K scores contains the local regions in the image that are most relevant to the global visual features. Here, K is used to control the number of regions that the model focuses on. Normalize the Top-K scores through the Softmax function to ensure that the sum of the association weights of all selected local regions is 1, enabling the model to learn the importance of different local regions relative to the global context. Multiply the global visual features by the association weights and sum the weighted features of all local regions to obtain the local context features. In this embodiment, this method improves the computational efficiency while focusing more on the key association information through sparsification and key information screening, and is applicable to scenarios with limited computational resources or where rapid acquisition of key context information is required.

[0090] In one embodiment, after obtaining the local context features by interacting the local fine-grained visual features and the global visual features based on the attention mechanism, it further includes:

[0091] Centered on each local fine-grained visual feature, extract the adjacent local fine-grained visual features within a preset sliding window through the preset sliding window to generate a local context candidate set;

[0092] Based on the multi-head attention mechanism, calculate the association scores between each local fine-grained visual feature and the corresponding local context candidate set;

[0093] Add the association scores to the corresponding local fine-grained visual features to generate local enhanced features, and use the local enhanced features as the updated representation of the local context features.

[0094] Specifically, the sliding window is a commonly used image processing technique for moving on an image and extracting a region of a fixed size. In this embodiment, the sliding window moves on the feature map and extracts adjacent local fine-grained visual features to obtain a local context candidate set. This candidate set contains all the local fine-grained visual features adjacent to the current local fine-grained visual feature, which is used to capture the context information of the local features. The multi-head attention mechanism is a technique extended from the self-attention mechanism, which can capture different context relationships by calculating multiple attention heads in parallel. Through the multi-head attention mechanism, multiple different context relationships can be attentively considered in parallel, so as to more comprehensively understand the context of local features and calculate the association scores between the local fine-grained visual features and each feature in the local context candidate set. By adding the association scores to the corresponding local fine-grained visual features, local enhanced features can be obtained. Different from generating local context features by weighted summation of global visual features, updating the local enhanced features to local context features in this embodiment focuses more on the interaction between local features and neighborhood information, and is applicable to application scenarios that require identifying key local features in the global context.

[0095] Based on the same inventive concept, as Figure 2 shown, the embodiment of the present application also provides a multi-modal understanding optimization system 200 based on fine-grained feature extraction and global information integration. The system includes:

[0096] An image feature extraction module 201, configured to divide an input image into multiple local image patches, and based on a pre-trained model, extract local fine-grained visual features of each local image patch and the global visual feature of the input image;

[0097] A global feature fusion module 202, configured to:

[0098] Based on the attention mechanism, interact the local fine-grained visual features with the global visual features to obtain local context features;

[0099] Fuse the local fine-grained visual features and the corresponding local context features to generate fused visual features;

[0100] A dynamic feature selection module 203, configured to map the fused visual features to the same semantic space as the text features of the large language model to obtain projected visual features, and based on the input text features, dynamically screen the projected visual features through attention weights to obtain key visual features;

[0101] An image semantic parsing module 204, configured to fuse the key visual features with the text features to generate a joint feature representation, and input the joint feature representation into the large language model to generate a semantic parsing result.

[0102] Specifically, the image feature extraction module 201 extracts the local fine-grained visual features of each local image patch and the global visual feature of the input image by splitting the input image into multiple local image patches, providing the details and overall information of the image for subsequent feature fusion and semantic parsing. The global feature fusion module 202 then introduces global information to enable the interaction between the local fine-grained visual features and the global visual features, thereby obtaining local context features, and fuses the local fine-grained visual features and the corresponding local context features to generate fused visual features, enhancing the subsequent model's ability to capture fine-grained target context information and effectively solving the problems of object and connection area segmentation and ignoring overall context information caused by traditional cropping strategies. The dynamic feature selection module 203 then performs feature screening based on the attention mechanism, combining the fused visual features and text features to remove redundant features and obtain key visual features, greatly reducing the computational overhead of the subsequent model and improving the running efficiency of the model. The image semantic parsing module 204 can then fuse the obtained key visual features and text features to generate a joint feature representation and input it into the large language model for semantic parsing. Through the coordinated operation of the above modules, the technical effect of this system is to construct an intelligent system that can deeply understand multi-modal content for various multi-modal application scenarios, realizing the efficient parsing, accurate recognition, and intelligent interaction of complex multi-modal information.

[0103] Furthermore, the global feature fusion module 202 includes a first local context feature extraction subunit for:

[0104] Using the local fine-grained visual feature as the query vector, the global visual feature as the key vector and value vector, and calculating the attention score;

[0105] Normalizing the attention score to generate the normalized first attention weight;

[0106] Fusing the normalized first attention weight with the global visual feature to obtain the local context feature.

[0107] Furthermore, the dynamic feature selection module 203 includes a dynamic screening feature subunit for:

[0108] Concatenating the projected visual feature and the global visual feature to obtain a concatenated feature;

[0109] Adding positional encodings to the concatenated feature and the text feature respectively to obtain the encoded concatenated feature and the encoded text feature;

[0110] Calculating the second attention weight between the encoded concatenated feature and the encoded text feature based on the attention mechanism of the large language model to generate an attention weight map;

[0111] Dynamically select visual features with a second attention weight higher than a preset threshold based on a preset threshold or weight sorting rule to obtain key visual features.

[0112] Further, calculate the local context feature through the following formula:

[0113]

[0114] c i = softmax(attn)·g

[0115] where attn is the attention score, g is the global visual feature, and d i is the visual feature representation of the i-th local image patch, i.e., the local fine-grained visual feature, and d k is the dimension of the feature vector, and c i is the local context feature of the i-th local image patch.

[0116] Further, calculate the second attention weight through the following formula:

[0117]

[0118] where attn' is the second attention weight, PE(*) is the position encoding function, t is the text feature, V' is the projected visual feature, g is the global visual feature, Concat(*) is used to concatenate two or more vectors within the parentheses, and d k is the dimension of the feature vector.

[0119] Further, the global feature fusion module 202 may further include a second local context feature extraction sub-unit for:

[0120] Based on the local fine-grained visual feature and the global visual feature, calculate the initial attention scores of each local fine-grained visual feature and the global visual feature;

[0121] Sparsify the initial attention scores corresponding to each local fine-grained visual feature, and filter out the set of region indices corresponding to the Top-K scores. The Top-K scores are the top K initial attention scores when sorting the initial attention scores from high to low. Here, K is a preset sparsity parameter;

[0122] Perform Softmax normalization on the Top-K scores to generate the association weights between the local fine-grained visual features and the global visual feature;

[0123] Based on the association weights, perform weighted summation on the global visual feature to generate the local context feature.

[0124] Further, the system further includes a local context feature enhancement module for:

[0125] Centering on each local fine-grained visual feature, extracting adjacent local fine-grained visual features within a preset sliding window through the preset sliding window to generate a local context candidate set;

[0126] Based on the multi-head attention mechanism, calculating the association scores between each local fine-grained visual feature and the corresponding local context candidate set;

[0127] Adding the association scores to the corresponding local fine-grained visual features to generate local enhanced features, and using the local enhanced features as the updated representation of the local context features.

[0128] In an exemplary embodiment, the present invention further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the multi-modal understanding optimization method based on fine-grained feature extraction and global information integration of the present application. A multi-core processor is preferably used to improve the parallel processing ability of the system. Memory: Provide sufficient temporary storage space to support the operation of the program and the processing of data. The memory capacity should be large enough to accommodate a large amount of supply information and computing tasks.

[0129] In an exemplary embodiment, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the multi-modal understanding optimization method based on fine-grained feature extraction and global information integration of the present application. The computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), solid-state drive (SSD, SolidState Drives), or optical disc, etc. Among them, the random access memory may include resistive random access memory (ReRAM, Resistance Random Access Memory) and dynamic random access memory (DRAM, Dynamic RandomAccess Memory).

[0130] The above embodiments only represent several implementation manners of the embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the embodiments of the application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A multimodal understanding optimization method based on fine-grained feature extraction and global information integration, characterized in that The method includes: Segmenting the input image into multiple local image patches, and based on a pre-trained model, extracting the local fine-grained visual features of each local image patch and the global visual feature of the input image; Based on the attention mechanism, interacting the local fine-grained visual features with the global visual feature to obtain local context features; Fusing the local fine-grained visual features and the corresponding local context features to generate fused visual features; Mapping the fused visual features to the same semantic space as the text features of the large language model to obtain projected visual features, and based on the input text features, dynamically screening the projected visual features through attention weights to obtain key visual features; Fusing the key visual features with the text features to generate a joint feature representation, and inputting the joint feature representation into the large language model to generate a semantic parsing result.

2. The method according to claim 1, characterized in that, The step of, based on the attention mechanism, interacting the local fine-grained visual features with the global visual feature to obtain local context features includes: Taking the local fine-grained visual features as query vectors, taking the global visual features as key vectors and value vectors, and calculating attention scores; Performing normalization processing on the attention scores to generate normalized first attention weights; Fusing the normalized first attention weights with the global visual features to obtain the local context features.

3. The method according to claim 1, characterized in that, The step of, based on the input text features, dynamically screening the projected visual features through attention weights to obtain key visual features includes: Concatenating the projected visual features with the global visual features to obtain concatenated features; Adding position encodings to the concatenated features and the text features respectively to obtain encoded concatenated features and encoded text features; Based on the attention mechanism of the large language model, calculating the second attention weights between the encoded concatenated features and the encoded text features to generate an attention weight map; According to a preset threshold or weight sorting rule, dynamically selecting, based on the attention weight map, the visual features with the second attention weights higher than the preset threshold to obtain key visual features.

4. The method according to claim 2, wherein The local context features are calculated by the following formula: c i = softmax(attn) · g where attn is the attention score, g is the global visual feature, and d i is the visual feature representation of the i-th local image patch, i.e., the local fine-grained visual feature, and d k is the dimension of the feature vector, and c i is the local context feature of the i-th local image patch.

5. The method according to claim 3, characterized in that, The second attention weights are calculated by the following formula: Among them, attn' is the second attention weight, PE(*) is the position encoding function, t is the text feature, V' is the projected visual feature, g is the global visual feature, Concat(*) is used to concatenate two or more vectors within the parentheses, and d k is the dimension of the feature vector.

6. The method according to claim 1, characterized in that The step of, based on the attention mechanism, interacting the local fine-grained visual features with the global visual feature to obtain local context features includes: Based on the local fine-grained visual features and the global visual features, calculating the initial attention scores of each local fine-grained visual feature and the global visual feature; Performing sparsification processing on the initial attention scores corresponding to each local fine-grained visual feature, and screening out the set of region indices corresponding to the Top-K scores, where the Top-K scores are the top K initial attention scores when the initial attention scores are sorted from high to low, and K is a preset sparsity parameter; Perform Softmax normalization on the Top-K scores to generate the association weights between the local fine-grained visual features and the global visual features; Perform weighted summation on the global visual features based on the association weights to generate the local context features.

7. The method according to claim 1, wherein After obtaining the local context features by interacting the local fine-grained visual features and the global visual features based on the attention mechanism, it further includes: Centered on each of the local fine-grained visual features, extract the adjacent local fine-grained visual features within the preset sliding window through the preset sliding window to generate a local context candidate set; Based on the multi-head attention mechanism, calculate the association scores between each of the local fine-grained visual features and the corresponding local context candidate set; Add the association scores to the corresponding local fine-grained visual features to generate local enhanced features, and use the local enhanced features as the updated representation of the local context features.

8. A multimodal understanding optimization system based on fine-grained feature extraction and global information integration, characterized in that, The system includes: An image feature extraction module, configured to divide the input image into multiple local image patches, and based on a pre-trained model, extract the local fine-grained visual features of each local image patch and the global visual feature of the input image; A global feature fusion module, configured to: Based on the attention mechanism, interact the local fine-grained visual features and the global visual features to obtain local context features; Fuse the local fine-grained visual features and the corresponding local context features to generate fused visual features; A dynamic feature selection module, configured to map the fused visual features to the same semantic space as the text features of the large language model to obtain the projected visual features, and dynamically filter the projected visual features based on the input text features to obtain key visual features; An image semantic parsing module, configured to fuse the key visual features and the text features to generate a joint feature representation, and input the joint feature representation into the large language model to generate a semantic parsing result.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images