Multi-level visual feature dynamic fusion method, device, equipment and medium

Abstract

The invention discloses a multi-level visual feature dynamic fusion method and device, equipment and a medium, and the method comprises the steps: introducing a task-perceived and content-driven multi-level feature fusion mechanism into a visual language large model, and carrying out the multi-level feature fusion according to different contents of an input image and a text instruction. Fusion weights are adaptively distributed to all layers of features of a visual encoder, the comprehensive understanding ability of the model for image details and global semantics is remarkably improved, and the application range of the visual language large model in fine-grained perception and diversified scenes is expanded, that is, by introducing a dynamic weight distribution mechanism, the visual language large model can be more accurately identified. Adaptive integration of different levels of visual features is realized, and the comprehensive understanding ability of a visual language large model for image details and global semantics is effectively improved.

Description

A method, apparatus, device and medium for dynamic fusion of multi-level visual features Technical Field

[0001] This invention relates to the field of visual large model technology, and in particular, to a method, apparatus, device and medium for dynamic fusion of multi-level visual features. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Vision-Language Models (LVLMs) have become a research hotspot in the field of multimodal intelligence. These models can comprehensively process visual and textual information to achieve complex tasks such as image description, visual question answering, and multimodal reasoning.

[0003] Currently, mainstream large-scale visual language models generally adopt the following basic architecture: three core modules, namely a visual encoder, a visual language adapter, and a large language model. The visual encoder (such as CLIP-ViT) extracts visual features from the input image; the visual language adapter (such as MLP and Q-former) projects and aligns the visual features to the input space of the large language model (such as Vicuna and LLaMA); and the large language model integrates text instructions and visual features, achieving task-driven text generation through large-scale pre-trained knowledge. This process provides the foundation for the widespread application of these models in various visual language tasks.

[0004] However, in current mainstream LVLM systems, the visual features extracted by the visual encoder often only come from the last layer (or penultimate layer) of CLIP-ViT. Although these high-level features have rich abstract semantics and can better meet the alignment requirements of large language models for visual semantics, their ability to perceive fine-grained visual information is limited. Specifically, this manifests in the following ways: 1) Loss of detail: While expressing abstract semantics, deep features often lose microscopic details in the image, such as local textures of objects and subtle structural differences, resulting in insufficient performance of the model for problems requiring detailed observation (such as local attribute recognition and local object relationship reasoning); 2) Single-level problem: Existing technologies only utilize features from a single level (such as the last layer), failing to fully leverage the complementary advantages of different levels of features in the visual encoder. Shallow features focus more on capturing low-level details of the image, while high-level features focus on global semantic information. Combining the two helps improve the model's comprehensive perception ability; 3) Static fusion problem: Some attempts attempt to simply splice or average multi-level features, but the static and fixed fusion method cannot adaptively adjust the feature fusion strategy according to the actual task or input content, resulting in insufficient flexibility and adaptability.

[0005] The aforementioned problems have gradually become apparent in the practical applications of large-scale visual language models. For example, in scenarios involving rich details (such as medical image analysis, industrial quality inspection, and complex question answering), the models often perform poorly due to a lack of fine-grained features. At the same time, different tasks have significantly different requirements for the level of visual features: some tasks focus on global semantic reasoning, while others rely on capturing local details, and existing methods are unable to dynamically adapt to these diverse needs. Summary of the Invention

[0006] This application provides a multi-level visual feature dynamic fusion method to address the technical problems of existing large visual language models in terms of fine-grained visual understanding, multi-level feature fusion, and dynamic adaptability.

[0007] This application achieves its goal through the following scheme: a multi-level visual feature dynamic fusion method, comprising the following steps: S1, receiving the original input image through the visual encoder of a large visual language model, sequentially inputting it into a visual encoding network composed of K Transformer layers, extracting the visual features of the image at each layer as the image patch representation output by each layer, and finally outputting a global representation for global semantic expression; S2, receiving text input through the text encoder of a large visual language model, and outputting a global embedding vector of the text through a text encoding network to express the semantic content of the text; S3, integrating visual and textual information, concatenating the global representation with the global embedding vector to obtain a joint feature vector for implementation. The process involves cross-modal information fusion and task guidance; S4: Based on the joint feature vector, a dynamic weight vector with a length equal to the number of Transformer layers K is obtained to predict the fusion weight of each layer's visual features, achieving task-adaptive feature level selection; S5: The visual features of each layer and the corresponding weights in the dynamic weight vector are weighted and summed to obtain the fused multi-level visual features; S6: The multi-level visual features are spatially mapped and dimensionally transformed to obtain adapted multi-level visual features that meet the input requirements of the large language model; S7: The adapted multi-level visual features and text features are input into the large language model, and based on the knowledge and reasoning capabilities of the large language model, the final text response of the multimodal task is output.

[0008] Furthermore, in step S1, the visual encoder includes a CLIP visual encoder and an OpenVision visual encoder.

[0009] Furthermore, in step S2, the text encoder includes CLIP text encoder and OpenVision text encoder.

[0010] Further, step S4 specifically includes the following steps: S41, receiving a joint feature vector obtained by concatenating the global representation and the global embedding vector; S42, inputting the joint feature vector into a two-layer multilayer perceptron, wherein the first layer of the multilayer perceptron uses a non-linear activation function and the second layer of the multilayer perceptron uses a softmax activation function; S43, outputting a normalized weight vector with a length equal to the number of Transformer layers K, used to predict the fusion weights of visual features at each layer, thereby achieving task-adaptive feature level selection.

[0011] Furthermore, in step S42, the nonlinear activation function includes the ReLU activation function and the GELU activation function.

[0012] Furthermore, in step S6, a multilayer perceptron and a Q-former are used to perform spatial mapping and dimensional transformation on the multilayer visual features to obtain adapted multilayer visual features.

[0013] Furthermore, in step S7, the large language model includes LLaMA and Vicuna.

[0014] This application also provides a multi-level visual feature dynamic fusion device, comprising: a visual feature extraction module, used to receive the original input image through the visual encoder of a large visual language model, and sequentially input it into a visual encoding network composed of K Transformer layers, extracting the visual features of the image at each layer, with each layer outputting image patch representations, and the final layer outputting a global representation for global semantic expression; a text encoding module, used to receive text input through the text encoder of a large visual language model, and output a global embedding vector of the text through a text encoding network to express the semantic content of the text; and a feature concatenation module, used to integrate visual and textual information, concatenating the global representation with the global embedding vector to obtain a joint feature vector, used to achieve cross-modal information fusion. The system includes: a task guidance module; a weight allocation module, used to predict the fusion weights of visual features at each layer by obtaining a dynamic weight vector of length equal to the number of Transformer layers K based on the joint feature vector, thus achieving task-adaptive feature layer selection; a layer-by-layer weighted summation module, used to weight and sum the visual features of each layer and the corresponding weights in the dynamic weight vector to obtain fused multi-level visual features; a large language model adaptation module, used to perform spatial mapping and dimensional transformation on the multi-level visual features to obtain adapted multi-level visual features that meet the input requirements of the large language model; and a text response output module, used to input the adapted multi-level visual features and text features into the large language model, and output the final text response of the multimodal task based on the knowledge and reasoning capabilities of the large language model.

[0015] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-level visual feature dynamic fusion method.

[0016] This application also provides a storage medium including a stored program that, when the program is executed, controls the device where the storage medium is located to perform the steps of the multi-level visual feature dynamic fusion method.

[0017] Compared with existing technologies, this application has the following beneficial effects: This application provides a multi-level visual feature dynamic fusion method, apparatus, device, and medium. The multi-level visual feature dynamic fusion method introduces a task-aware, content-driven multi-level feature fusion mechanism into the Visual Language Model (LVLM). It can adaptively assign fusion weights to the features of each layer of the visual encoder according to the different contents of the input image and text instructions, significantly improving the model's comprehensive understanding of image details and global semantics. It expands the application scope of the Visual Language Model in fine-grained perception and diverse scenarios. That is, by introducing a dynamic weight allocation mechanism, adaptive integration of visual features at different levels is achieved, effectively improving the Visual Language Model's comprehensive understanding of image details and global semantics.

[0018] In addition to the purposes, features, and advantages described above, this application has other purposes, features, and advantages. A further detailed description of this application will be provided below with reference to the figures. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them: Figure 1 is a flowchart of the multi-level visual feature dynamic fusion method of the preferred embodiment of this application; Figure 2 is a flowchart of the sub-step of step S4 of the preferred embodiment of this application; Figure 3 is a schematic diagram of the multi-level visual feature dynamic fusion device module of the preferred embodiment of this application; Figure 4 is a schematic diagram of the multi-level visual feature dynamic fusion system module of the preferred embodiment of this application; Figure 5 is a block diagram of the electronic device of the preferred embodiment of this application; Figure 6 is an internal structure diagram of the computer device of the preferred embodiment of this application. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a multi-level visual feature dynamic fusion device capable of achieving the above functions. The following description uses a multi-level visual feature dynamic fusion device as the executing entity to illustrate this embodiment and the subsequent embodiments.

[0024] As shown in Figure 1, a preferred embodiment of this application provides a multi-level visual feature dynamic fusion method, including the following steps: S1, receiving the original input image through the visual encoder of a large visual language model, sequentially inputting it into a visual encoding network composed of K Transformer layers, extracting the visual features of the image at each layer as the image patch representation output by each layer, and finally outputting a global representation for global semantic expression. The image patch representation of each layer serves as a candidate feature for subsequent multi-level fusion; S2, receiving text input through the text encoder of a large visual language model, and outputting a global embedding vector of the text through a text encoding network to express the semantic content of the text; S3, integrating visual and textual information, and performing a process of combining the global representation with the global embedding vector. S4. By concatenating the rows, a joint feature vector is obtained, which is used to achieve cross-modal information fusion and task guidance; S5. Based on the joint feature vector, a dynamic weight vector with a length equal to the number of Transformer layers K is obtained to predict the fusion weight of each layer's visual features, realizing task-adaptive feature level selection; S6. The visual features of each layer and the corresponding weights in the dynamic weight vector are weighted and summed to obtain the fused multi-level visual features; S7. The multi-level visual features are spatially mapped and dimensionally transformed to obtain adapted multi-level visual features, making them suitable for the input requirements of the large language model; S8. The adapted multi-level visual features and text features are input into the large language model, and based on the knowledge and reasoning ability of the large language model, the final text response of the multimodal task is output.

[0025] The multi-level visual feature dynamic fusion method in this embodiment introduces a task-aware, content-driven multi-level feature fusion mechanism into the Visual Language Model (LVLM). It can adaptively assign fusion weights to the features of each layer of the visual encoder according to the different content of the input image and text instructions, significantly improving the model's comprehensive understanding of image details and global semantics. This expands the application scope of the Visual Language Model in fine-grained perception and diverse scenarios. In other words, by introducing a dynamic weight allocation mechanism, it achieves adaptive integration of visual features at different levels, effectively improving the Visual Language Model's comprehensive understanding of image details and global semantics.

[0026] Preferably, in step S1, the visual encoder is a CLIP visual encoder or an OpenVision visual encoder.

[0027] Preferably, in step S2, the text encoder is a CLIP text encoder or an OpenVision text encoder.

[0028] As shown in Figure 2, in a preferred embodiment of this application, step S4 specifically includes the following steps: S41, receiving a joint feature vector obtained by concatenating the global representation and the global embedding vector; S42, inputting the joint feature vector into a two-layer multilayer perceptron, wherein the first layer of the multilayer perceptron uses a nonlinear activation function and the second layer of the multilayer perceptron uses a softmax activation function, and the nonlinear activation function uses ReLU activation function and GELU activation function; S43, outputting a normalized weight vector with a length equal to the number of Transformer layers K, used to predict the fusion weight of visual features in each layer, thereby achieving task-adaptive feature level selection.

[0029] This embodiment employs a "two-layer MLP structure" for weight allocation, where the first layer uses a non-linear activation function (such as ReLU or GELU), and the second layer uses a softmax activation function. The design principle is as follows: the non-linear activation of the first layer enhances the network's ability to model the complex relationships between input image and text features, ensuring that weight allocation is not limited to simple linear combinations; while the softmax normalization of the second layer guarantees that the output weights are positive and sum to 1, giving the weights assigned to each layer of visual features good physical interpretability and numerical stability. This design effectively achieves adaptive and dynamic generation of multi-level feature fusion weights, meeting the diverse needs of complex tasks while ensuring the rationality of feature fusion and the system's training convergence.

[0030] Preferably, in step S6, a multilayer perceptron or Q-former is used to perform spatial mapping and dimensional transformation on the multilayer visual features to obtain adapted multilayer visual features.

[0031] Preferably, in step S7, the large language model is LLaMA or Vicuna.

[0032] As shown in Figure 3, another preferred embodiment of this application also provides a multi-level visual feature dynamic fusion device, including: a visual feature extraction module, used to receive the original input image through the visual encoder of a large visual language model, and input it sequentially into a visual encoding network composed of K Transformer layers, extracting the visual features of the image at each layer, each layer can output image patch representations, and the final layer outputs a global representation for global semantic expression; a text encoding module, used to receive text input through the text encoder of a large visual language model, and output a global embedding vector of the text through a text encoding network to express the semantic content of the text; and a feature concatenation module, used to integrate visual and text information, concatenating the global representation with the global embedding vector to obtain a joint feature vector, used to achieve cross-modal processing. The system comprises: a dynamic weighting module for information fusion and task guidance; a weight allocation module for predicting the fusion weights of visual features at each layer based on the joint feature vector and a dynamic weight vector of length equal to the number of Transformer layers K, thus enabling adaptive feature layer selection for the task; a weighted summation module for weighted summation of the visual features of each layer and the corresponding weights in the dynamic weight vector to obtain fused multi-level visual features; a large language model adaptation module for spatial mapping and dimensional transformation of the multi-level visual features to obtain adapted multi-level visual features that meet the input requirements of the large language model; and a text response output module for inputting the adapted multi-level visual features and text features into the large language model and, based on the knowledge and reasoning capabilities of the large language model, outputting the final text response for the multimodal task.

[0033] The multi-level visual feature dynamic fusion device provided in this application, employing the multi-level visual feature dynamic fusion method in the above embodiments, can solve the technical problems of insufficient fine-grained visual understanding, multi-level feature fusion, and dynamic adaptability of existing large visual language models. Compared with the prior art, the beneficial effects of the multi-level visual feature dynamic fusion device provided in this application are the same as those of the multi-level visual feature dynamic fusion method provided in the above embodiments, and other technical features in the multi-level visual feature dynamic fusion device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0034] Another preferred embodiment of this application provides a multi-level visual feature dynamic fusion system for large visual language models, the system structure of which is shown in Figure 4. The system mainly includes the following functional modules: a CLIP visual encoder module, a CLIP text encoder module, a feature splicing module (including a multimodal guiding signal unit), a weight allocator module, an inter-layer weighted summation module, a visual language adapter module, and a large language model module. Specifically, the CLIP visual encoder module receives the original input image through the CLIP visual encoder, sequentially inputs it into a visual encoding network composed of K Transformer layers, extracts the visual features of the image at each layer, outputs image patch representations at each layer, and finally outputs a global representation for global semantic expression.

[0035] The CLIP text encoder module is used to receive text input through the CLIP text encoder, and output a global embedding vector of the text through a text encoding network to express the semantic content of the text.

[0036] The feature splicing module (including the multimodal guidance signal unit) is used to integrate visual and textual information, splicing the global representation with the global embedding vector to obtain a joint feature vector, which is used to realize cross-modal information fusion and task guidance.

[0037] The weight allocator module is used to predict the fusion weight of each visual feature based on the joint feature vector, which is a dynamic weight vector with a length equal to the number of Transformer layers K, thereby enabling task-adaptive feature layer selection.

[0038] The inter-layer weighted summation module is used to sum the visual features of each layer and the corresponding weights in the dynamic weight vector to obtain the fused multi-layer visual features.

[0039] The visual language adapter module is used to perform spatial mapping and dimensional transformation on the multi-level visual features to obtain adapted multi-level visual features, so that they can be adapted to the input requirements of a large language model.

[0040] The large language model module inputs the adapted multi-level visual features and text features into the large language model, and outputs the final text response for the multimodal task based on the knowledge and reasoning ability of the large language model.

[0041] In this embodiment, the multi-level visual feature dynamic fusion system takes an outdoor street scene image and the text question "What color is the flag on the balcony?" as input. The image is processed by the CLIP visual encoder module to extract K layers of visual features (including patch features and CLS token features). The text question is processed by the CLIP text encoder module to obtain a sentence-level feature vector. The final layer CLS token features are concatenated with the sentence features and input into a two-layer MLP (the first layer uses ReLU activation function, and the second layer uses softmax), outputting a weight vector of length K. This weight is used to weight and sum the K layers of visual features to obtain the fused multi-level visual features. The fused features are mapped by an adapter and input into a large language model along with the text question, ultimately generating the response text: "The flag is red and white."

[0042] In addition, the weight allocator module, as a new addition in this embodiment, is embedded between the CLIP visual encoder module and the inter-level weighted summation module. Its input is connected to the feature concatenation module, and its output is connected to the inter-level weighted summation module. The module is designed as a pluggable structure, allowing direct integration into existing large-scale visual language model systems without disrupting the core structure of the original model. The inter-level weighted summation module, combined with the output of the weight allocator module, enables adaptive dynamic fusion of hierarchical features, providing the model with richer and finer-grained visual representations.

[0043] This application introduces a dynamic weight allocation mechanism to achieve adaptive integration of visual features at different levels, effectively improving the visual language large model's comprehensive understanding of image details and global semantics, and has the following beneficial technical effects: (1) Multi-level feature complementarity enhances model expressive ability. The shallow features of the visual encoder are mainly responsible for capturing the low-level information of the image, such as edges, textures and local structures, while the deep features are better at representing global semantics and abstract concepts. The existing visual language large model LVLM only uses the last layer of features, which limits the model's performance when dealing with fine-grained tasks. This application can fully explore the complementarity of the two by dynamically fusing shallow and deep features, so that the model can pay attention to local details and maintain global consistency, which theoretically improves the model's ability to express multi-scale information; (2) Weight allocator realizes task-adaptive feature fusion. Traditional methods use static fusion strategies (such as simple splicing or averaging), which cannot adaptively adjust the weight allocation of feature levels for different tasks or inputs. This application uses a weight allocator module to dynamically calculate the optimal fusion weight based on the joint representation of the input image and text content. In theory, this mechanism can allocate the "bandwidth" of the information path according to the needs of the task, enhance the model's ability to balance between fine-grained and global semantics, and improve the model's generalization and adaptability; (3) The gradient is transferable, and the end-to-end optimization weight allocator module is based on the differentiable MLP and softmax structure. Its weight allocation process is completely differentiable to the downstream loss function, and it can be trained in collaboration with the entire visual language model to achieve end-to-end global optimal optimization and improve the overall performance of the model; (4) Experiments on the prototype system on public datasets show that this application has achieved significant performance improvements in the following typical multimodal tasks: Visual Question Answering (VQA) task: The accuracy of the model of this invention in detail-sensitive questions such as fine-grained question answering, attribute recognition, and local region reasoning is 3-8 percentage points higher than that of the baseline model, especially in question answering tasks that require capturing the microstructure of images.

[0044] Image captioning task: After multi-level dynamic fusion, the generated descriptions are more detailed and richer, covering more details and context-related information. Both subjective evaluation and automatic evaluation metrics are superior to existing solutions.

[0045] Complex scene reasoning tasks: In scenarios with multiple objects, occlusion, or complex relationships, the model of this invention can effectively utilize information from shallow and deep features to achieve comprehensive reasoning on fine-grained semantics and global semantics, thereby improving task robustness.

[0046] Performance Comparison with Existing Technologies: Compared to existing methods that only employ a single high-level visual feature or static multi-layer fusion, this application can: adapt to different inputs (the input of a large visual language model typically includes two parts: 1) an image; 2) a text instruction or question about the image. Therefore, in the example case, the input includes both the text question and the image itself. When performing multi-level feature fusion, our method simultaneously considers these two parts of input information (e.g., image information in the CLS feature and text information extracted by the CLIP text encoder) and adaptively adjusts the feature level weights, improving the model's fine-grained understanding and reasoning capabilities; significantly enhancing the model's generalization and adaptability to complex and detail-rich tasks; and maintaining the system's end-to-end training capability, facilitating engineering implementation and large-scale deployment.

[0047] Specifically, this application introduces an input-adaptive weight allocation mechanism, enabling the fusion weights of multi-level visual features to be dynamically adjusted based on the content of the input image and text instructions. Specifically, the weight allocator module generates the optimal fusion weight distribution in real time based on the current input global image features and text semantics, thereby selectively highlighting visual information at different levels. This technique yields the following results: First, for fine-grained perception tasks (such as local attribute recognition and small object detection), the model can automatically increase the fusion weights of shallow features based on task-related signals, ensuring sufficient preservation of low-level details in the image. Simultaneously, in tasks requiring global semantic understanding, the weight allocator can appropriately increase the weights of deep features, making the model more focused on high-level semantic abstraction. Through this content-aware hierarchical feature scheduling, the model can flexibly balance details and semantic information, achieving comprehensive adaptation to both fine-grained and global tasks. Therefore, the model exhibits stronger understanding and reasoning abilities in tasks sensitive to detail and complex reasoning.

[0048] Secondly, the dynamic fusion mechanism effectively improves the model's adaptability and generalization ability to input changes. Because the model can adaptively select the most suitable feature fusion strategy based on different input types without requiring manual specification or preset weights, it can maintain excellent performance in a wider range of complex tasks and detailed scenarios. This reduces dependence on a single feature level and enhances the model's robustness and generalization ability when facing unseen tasks and new scenarios.

[0049] Finally, the proposed weight allocator employs a differentiable MLP structure and is jointly trained end-to-end with the main model, without introducing complex post-processing or external modules, ensuring efficient end-to-end optimization of the entire system. This not only facilitates system engineering implementation and deployment but also enables rapid integration and promotion in large-scale data and application environments, significantly enhancing the model's practicality and engineering feasibility.

[0050] In summary, this application fully utilizes the multi-level feature potential of visual encoders to realize a large visual language model with high adaptability, high performance and strong fine-grained perception capabilities, which has outstanding technical innovation and practical application value.

[0051] As shown in Figure 5, a preferred embodiment of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the multi-level visual feature dynamic fusion method in the above embodiments.

[0052] The electronic device provided in this application employs the multi-level visual feature dynamic fusion method described in the above embodiments, which can solve the technical problems of insufficient fine-grained visual understanding, multi-level feature fusion, and dynamic adaptability of existing large visual language models. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the multi-level visual feature dynamic fusion method provided in the above embodiments, and other technical features in the electronic device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0053] As shown in Figure 6, a preferred embodiment of this application also provides a computer device, which can be a terminal or a liveness detection server, and its internal structure diagram is shown in Figure 6. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with other external computer devices via a network connection. When the computer program is executed by the processor, it implements the steps of the above-described multi-level visual feature dynamic fusion method.

[0054] Those skilled in the art will understand that the structure shown in Figure 6 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0055] The computer device provided in this application, employing the multi-level visual feature dynamic fusion method described in the above embodiments, can solve the technical problems of insufficient fine-grained visual understanding, multi-level feature fusion, and dynamic adaptability of existing large visual language models. Compared with the prior art, the beneficial effects of the computer device provided in this application are the same as those of the multi-level visual feature dynamic fusion method provided in the above embodiments, and other technical features in the electronic device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0056] A preferred embodiment of this application also provides a storage medium, the storage medium including a stored program, which, when the program is executed, controls the device where the storage medium is located to perform the steps of the multi-level visual feature dynamic fusion method in the above embodiments.

[0057] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0058] If the functions described in this embodiment are implemented as software functional units and sold or used as independent products, they can be stored in one or more computing device-readable storage media. Based on this understanding, the parts of this application's embodiments that contribute to the prior art or the technical solutions can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computer, server, mobile computing device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0059] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0060] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0063] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multi-level visual feature dynamic fusion method described above.

[0064] The computer program product provided in this application can solve the technical problems of the shortcomings of existing large visual language models in terms of fine-grained visual understanding, multi-level feature fusion, and dynamic adaptability. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multi-level visual feature dynamic fusion method provided in the above embodiments, and will not be repeated here.

[0065] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0066] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A multi-level visual feature dynamic fusion method, characterized in that, The steps include: S1. Receiving the original input image through the visual encoder of the visual language large-scale model, and sequentially inputting it into a visual encoding network composed of K Transformer layers, extracting the visual features of the image at each layer as the image patch representation output by each layer, and finally outputting a global representation for global semantic expression; S2. Receiving text input through the text encoder of the visual language large-scale model, and outputting a global embedding vector of the text through the text encoding network to express the semantic content of the text; S3. Integrating visual and textual information, concatenating the global representation with the global embedding vector to obtain a joint feature vector, which is used to realize cross-modal information fusion and task guidance; S4. Predicting the fusion weight of the visual features of each layer based on the joint feature vector and obtaining a dynamic weight vector with a length equal to the number of Transformer layers K, realizing task-adaptive feature level selection; S5. Weighted summation of the visual features of each layer and the corresponding weights in the dynamic weight vector is performed to obtain the fused multi-level visual features. S6. Perform spatial mapping and dimensional transformation on the multi-level visual features to obtain adapted multi-level visual features that are compatible with the input requirements of the large language model; S7. Input the adapted multi-level visual features and text features into the large language model, and output the final text response of the multimodal task based on the knowledge and reasoning ability of the large language model.

2. The multi-level visual feature dynamic fusion method according to claim 1, characterized in that, In step S1, the visual encoder includes CLIP visual encoder and OpenVision visual encoder.

3. The multi-level visual feature dynamic fusion method according to claim 1, characterized in that, In step S2, the text encoder includes CLIP text encoder and OpenVision text encoder.

4. The multi-level visual feature dynamic fusion method according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41, receiving a joint feature vector obtained by concatenating the global representation and the global embedding vector; S42, inputting the joint feature vector into a two-layer multilayer perceptron, wherein the first layer of the multilayer perceptron uses a non-linear activation function and the second layer of the multilayer perceptron uses a softmax activation function; S43, outputting a normalized weight vector with a length equal to the number of Transformer layers K, which is used to predict the fusion weight of visual features at each layer, thereby achieving task-adaptive feature level selection.

5. The multi-level visual feature dynamic fusion method according to claim 4, characterized in that, In step S42, the nonlinear activation function includes the ReLU activation function and the GELU activation function.

6. The multi-level visual feature dynamic fusion method according to claim 1, characterized in that, In step S6, a multilayer perceptron and a Q-former are used to perform spatial mapping and dimensional transformation on the multilayer visual features to obtain adapted multilayer visual features.

7. The multi-level visual feature dynamic fusion method according to claim 1, characterized in that, In step S7, the large language model includes LLaMA and Vicuna.

8. A multi-level visual feature dynamic fusion device, characterized in that, include: The visual feature extraction module receives the original input image through the visual encoder of the visual language large-scale model, and sequentially inputs it into a visual encoding network consisting of K Transformer layers. It extracts the visual features of the image at each layer, with each layer outputting an image patch representation. The final layer outputs a global representation for global semantic expression. The text encoding module receives text input through the text encoder of the visual language large-scale model, and outputs a global embedding vector of the text through a text encoding network to express the semantic content of the text. The feature concatenation module integrates visual and textual information by concatenating the global representation with the global embedding vector to obtain a joint feature vector, used for cross-modal information fusion and task guidance. The weight allocation module predicts the fusion weights of the visual features at each layer based on a dynamic weight vector of length K equal to the number of Transformer layers obtained from the joint feature vector, enabling task-adaptive feature level selection. The inter-layer weighted summation module is used to sum the visual features of each layer and the corresponding weights in the dynamic weight vector to obtain the fused multi-layer visual features. The large language model adaptation module is used to perform spatial mapping and dimensional transformation on the multi-level visual features to obtain adapted multi-level visual features, so that they are adapted to the input requirements of the large language model; the text response output module is used to input the adapted multi-level visual features and text features into the large language model, and output the final text response of the multimodal task based on the knowledge and reasoning ability of the large language model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multi-level visual feature dynamic fusion method as described in any one of claims 1 to 7.

10. A storage medium comprising a stored program that, when the program is executed, controls a device in which the storage medium resides to perform the steps of the multi-level visual feature dynamic fusion method as described in any one of claims 1 to 7.

Images