Data processing method, apparatus, device, computer program product, and storage medium

Abstract

A data processing method, device, equipment, computer program product and computer readable storage medium are provided. The method keeps a trained multi-modal large language model unchanged, and constructs an additional visual feature bypass for the multi-modal large language model, wherein the visual feature bypass includes a model for generating multi-scale image features for input image data and a feature injector having the same adapter layer structure as an adapter of the multi-modal large language model to inject the generated multi-scale image features into the adapter of the multi-modal large language model through the bypass, realize the fusion of original image features and multi-scale image features, and perform an image processing task of the large language model based on the fused image features, so that the original image-text alignment capability of the multi-modal large language model is maintained while the visual perception capability of the multi-modal large language model is enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a data processing method, apparatus, device, computer program product, and storage medium. Background Technology

[0002] In recent years, multimodal large language models have made significant progress in the field of vision-language joint understanding. A multimodal large language model typically consists of three components: an image encoder, an adapter, and a large language model (LLM). Specifically, in a multimodal large language model, the image encoder converts the image input to the model into an image token. The adapter maps the image token generated by the visual encoder to the feature dimensions of a text token, and then feeds the feature-mapped image token along with the language instruction input to the large language model. Visual perception enhancement techniques applied to multimodal large language models improve the model's visual feature extraction capabilities, enabling it to more accurately understand and analyze detailed information (such as object shape, texture, and spatial relationships) in the input image and efficiently align it with the language modality, thereby improving the model's overall performance in cross-modal tasks.

[0003] Existing methods for enhancing visual perception in multimodal large language models primarily achieve this by increasing the resolution of the input image or by combining different visual expert models in the visual encoder and performing pre-training and supervised fine-tuning. However, these methods still fail to effectively resolve the inherent contradiction between the underlying architecture of the visual encoder and the requirements of multimodal tasks: traditional image-text pre-training models emphasize high-level semantic associations while neglecting fine-grained features, while the combination of visual expert models undermines the pre-training advantages of the model in cross-modal alignment. This technical contradiction leads to significant performance bottlenecks in existing multimodal large language models in application scenarios that require simultaneous processing of fine visual features and complex semantic reasoning (such as industrial quality inspection and remote sensing image analysis).

[0004] Therefore, a visual perception enhancement method is needed that can effectively improve the extraction of image detail features while maintaining compatibility with existing multimodal large language model architectures, and at the same time ensure the alignment of visual-text feature spaces. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure proposes a visual perception enhancement method for multimodal large language models based on feature injection. By keeping the trained multimodal large language model unchanged, multi-scale image features of the input image are injected into the adapter of the multimodal large language model through feature bypass, thereby enhancing the visual perception capability of the multimodal large language model while preserving the original image-text alignment capability of the multimodal large model.

[0006] Embodiments of this disclosure provide a data processing method, apparatus, device, computer program product, and computer-readable storage medium.

[0007] Embodiments of this disclosure provide a data processing method, comprising: for acquired image data, generating a first image feature and a second image feature of the image data respectively through a visual encoder and a visual feature extraction model, wherein the second image feature is a multi-scale image feature of the image data; inputting the first image feature and the second image feature into a feature injector to generate injected features for each adapter layer of an adapter, wherein the feature injector includes the same adapter layer structure as the adapter; inputting the first image feature into the adapter and injecting injected features for each adapter layer of the adapter to generate an image token corresponding to the image data, wherein the generated image token has a feature dimension matching the input of a large language model; and inputting the image token into the large language model to perform a predetermined image processing task for the image data.

[0008] Embodiments of this disclosure provide a data processing apparatus, comprising: a feature extraction module configured to generate a first image feature and a second image feature of acquired image data using a visual encoder and a visual feature extraction model, respectively, wherein the second image feature is a multi-scale image feature of the image data; a feature generation module configured to input the first image feature and the second image feature into a feature injector to generate injected features for each adapter layer of an adapter, wherein the feature injector includes the same adapter layer structure as the adapter; a feature injection module configured to input the first image feature into the adapter and inject injected features for each adapter layer of the adapter to generate an image token corresponding to the image data, wherein the generated image token has a feature dimension matching the input of a large language model; and a task execution module configured to input the image token into the large language model to perform a predetermined image processing task for the image data.

[0009] Embodiments of this disclosure provide a data processing apparatus, including: one or more processors; and one or more memories, wherein the one or more memories store a computer-executable program, and when the processor executes the computer-executable program, it performs the data processing method as described above.

[0010] Embodiments of this disclosure provide a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, are used to implement the data processing method described above.

[0011] Embodiments of this disclosure provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a data processing method according to embodiments of this disclosure.

[0012] The method provided in the embodiments of this disclosure maintains the trained multimodal large language model unchanged by constructing an additional visual feature bypass for it. This visual feature bypass includes a model that generates multi-scale image features from the input image data and a feature injector with an adapter layer structure identical to the adapter of the multimodal large language model. The generated multi-scale image features are injected into the adapter of the multimodal large language model through this bypass, achieving the fusion of the original image-text alignment features and the multi-scale image features. This allows the image processing task of the large language model to be performed based on the fused image features. The method of the embodiments of this disclosure utilizes an additional visual feature bypass independent of the multimodal large language model to inject multi-scale image features layer by layer into the original image features based on the adapter layer structure. This enhances the visual perception capability of the multimodal large language model while maintaining its original image-text alignment capability. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some exemplary embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0014] Figure 1 This is a schematic diagram illustrating a scenario in which a response is generated using a pre-trained multimodal large language model according to an embodiment of the present disclosure;

[0015] Figure 2This is a schematic diagram illustrating visual perception enhancement for a pre-trained multimodal large language model;

[0016] Figure 3 This is a flowchart illustrating a data processing method according to an embodiment of the present disclosure;

[0017] Figure 4 This is a schematic diagram illustrating a data processing method according to an embodiment of the present disclosure;

[0018] Figure 5 This is a schematic diagram illustrating the injection of features into an adapter using a feature injector according to an embodiment of the present disclosure;

[0019] Figure 6 This is a schematic diagram illustrating a data processing apparatus according to an embodiment of the present disclosure;

[0020] Figure 7 A schematic diagram of a data processing apparatus according to an embodiment of the present disclosure is shown; and

[0021] Figure 8 A schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0023] In this specification and accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements are omitted. Furthermore, in the description of this disclosure, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0024] In embodiments of this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0026] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0027] The data processing method disclosed herein can be based on a Multimodal Large Language Model (MLLM). MLLM is a cutting-edge technology in the field of artificial intelligence that combines the characteristics of multimodal learning and large-scale pre-trained models. It can simultaneously process and understand multiple types of data (such as text, images, audio, and video) and achieve cross-modal information fusion and reasoning. Typical architectures of MLLMs, such as LLaVA, InternVL, and Qwen-VL, typically consist of three core components: an image encoder that converts the input image into high-dimensional visual features (image tokens), an adapter that maps the visual features to a text feature space compatible with the MLLM, and finally, the MLLM fuses visual and textual information to complete the reasoning. These models rely on pre-trained visual encoders (such as CLIP and SigLIP) to extract image semantics, and their training objective focuses on the global semantic alignment of images and text.

[0028] The data processing method disclosed herein is also based on visual perception enhancement techniques for multimodal large language models. Visual perception enhancement aims to improve the model's ability to capture image details (such as microstructure, texture features, spatial relationships, etc.) while maintaining the stability of cross-modal semantic alignment. The core objective of visual perception enhancement for multimodal large language models is to solve the problems of image detail loss and semantic association bias caused by the limitations of visual encoders in traditional multimodal large language models, ultimately achieving higher-precision visual-language joint reasoning (such as image description generation, visual question answering, fine-grained classification, and other tasks).

[0029] In summary, the solutions provided by the embodiments of this disclosure involve technologies such as multimodal large language models and their visual perception enhancement. The embodiments of this disclosure will be further described below with reference to the accompanying drawings.

[0030] Figure 1 This is a schematic diagram illustrating a scenario in which a response is generated using a pre-trained multimodal large language model according to an embodiment of the present disclosure.

[0031] like Figure 1As shown, in scenario 100, raw image data (such as natural scene images, medical images, industrial inspection images, etc.) can be received, and text data such as language prompts can be received simultaneously. The received image data and text data can be used as input to an MLLM to generate corresponding response outputs through a pre-trained MLLM. The MLLM can include a text-image aligned vision encoder, an adapter, and a large language model (LLM).

[0032] With the increasing application of multimodal large language models in fields such as industrial defect detection and medical image analysis, visual perception enhancement of multimodal large language models has become a key technical challenge. Visual perception enhancement aims to improve the model's ability to capture image details (such as microstructure, texture features, and spatial relationships) while maintaining the stability of cross-modal semantic alignment.

[0033] Currently, the mainstream visual perception enhancement methods can be mainly divided into two categories: the first type of visual perception enhancement method, which increases the resolution of the input image, and the second type of visual perception enhancement method, which combines different visual expert models in the visual encoder part and performs pre-training and supervised fine-tuning. Figure 2 This is a schematic diagram illustrating visual perception enhancement of a pre-trained multimodal large language model. Among them, Figure 2 (a) in the text corresponds to the first type of visual perception enhancement method, while Figure 2 (b) in the text corresponds to this second type of visual perception enhancement method. Figure 2 In this context, the image-text aligned visual encoder can employ a visual backbone network pre-trained with large-scale image-text pairs (such as CLIP-ViT or SigLIP-ResNet) to encode the input image into a high-dimensional visual feature vector. As a cross-modal feature fusion hub, the adapter contains learnable adapter layers (projection layers, such as linear transformations or lightweight MLPs) that map the image features output by the visual encoder to feature vectors with the same dimensionality as the text embedding space of a large language model. The LLM can be based on a pre-trained language model with a Transformer architecture (such as GPT-4 or LLaMA), receiving a concatenated input of image tokens output by the adapter and text tokens corresponding to the text data generated by a tokenizer. Through a self-attention mechanism, it achieves cross-modal joint inference, generating a natural language response related to the input command.

[0034] The first type of visual perception enhancement methods indirectly improve resolution by increasing the resolution of the input image or by using image slicing strategies, such as... Figure 2As shown in (a), the core of this approach lies in expanding the input dimension of the visual encoder. These methods commonly employ image-text alignment-based pre-trained visual encoders such as CLIP and SigLIP. While they perform well in large-scale image-text semantic association, they are limited by the inherent architecture of these pre-trained models—the training data and objective functions of encoders like CLIP are not optimized for multi-scale or low-level visual features (such as edges and corners), resulting in weaker ability to perceive image details. This limits the performance of large multimodal models using these pre-trained models as visual encoders in handling tasks requiring detailed visual analysis. This leads to a significant increase in computational resources required for high-resolution inputs, without a linear improvement in detail resolution. For example, when handling visual tasks requiring pixel-level resolution (such as fine-grained classification and medical image analysis), model performance significantly declines, and simply increasing resolution does little to improve model accuracy while requiring substantial additional computational resources.

[0035] The second type of visual perception enhancement method attempts to combine multiple visual expert models at the visual encoder to compensate for the shortcomings of a single encoder and improve visual perception capabilities, such as... Figure 2 As shown in (b) in the figure. Although such methods improve fine-grained recognition metrics on specific datasets and overcome to some extent the shortcomings of using a pre-trained image-text model as a visual encoder in image detail perception, they introduce two major new problems: First, the model needs to be re-pre-trained and supervised fine-tuned to integrate the features of the expert model, and the parameter knowledge of the existing large language model cannot be directly transferred, resulting in a significant increase in training costs; Second, since the added visual expert model lacks the ability to correlate text semantics, simply combining visual expert models often affects the alignment effect of the adapter's visual-text features, resulting in dimensionality mismatch in the visual-text features after adapter projection, thereby limiting the capabilities of the multimodal large language model.

[0036] Based on this, this disclosure provides a method for enhancing the visual perception of a multimodal large language model based on feature injection. By keeping the trained multimodal large language model unchanged, the multi-scale image features of the input image are injected into the adapter of the multimodal large language model through feature bypass, so as to enhance the visual perception capability of the multimodal large language model while retaining the original image-text alignment capability of the multimodal large model.

[0037] Optionally, the method provided in the embodiments of this disclosure can be implemented in a hardware environment consisting of high-performance computing devices and high-efficiency storage devices to collaboratively complete the inference task of a multimodal large model. The computing devices can employ GPUs (Graphics Processing Units) or TPUs (Tensor Processing Units) to leverage their massively parallel computing capabilities to accelerate the complex computations of deep learning models, significantly improving model training efficiency and real-time inference speed. The storage devices are responsible for storing model parameters, training datasets, and intermediate computation results, and are typically configured with high-speed solid-state drives (SSDs) or distributed storage systems to ensure rapid read and write of large-scale parameters.

[0038] The method provided in the embodiments of this disclosure maintains the trained multimodal large language model unchanged by constructing an additional visual feature bypass for it. This visual feature bypass includes a model that generates multi-scale image features from the input image data and a feature injector with an adapter layer structure identical to the adapter of the multimodal large language model. The generated multi-scale image features are injected into the adapter of the multimodal large language model through this bypass, achieving the fusion of the original image-text alignment features and the multi-scale image features. This allows the image processing task of the large language model to be performed based on the fused image features. The method of the embodiments of this disclosure utilizes an additional visual feature bypass independent of the multimodal large language model to inject multi-scale image features layer by layer into the original image features based on the adapter layer structure. This enhances the visual perception capability of the multimodal large language model while maintaining its original image-text alignment capability.

[0039] Figure 3 This is a flowchart illustrating a data processing method 300 according to an embodiment of the present disclosure. Figure 4 This is a schematic diagram illustrating a data processing method according to an embodiment of the present disclosure.

[0040] like Figure 4 As shown, the data processing method disclosed herein can be implemented through a feature-injection-based multimodal large language model visual perception enhancement framework. This framework includes two main parts: a pre-trained multimodal large language model and a visual feature bypass that is structurally independent of the multimodal large language model. The pre-trained multimodal large language model can utilize its learned image-text alignment capabilities to extract image features from the input image, while the visual feature bypass can extract more multi-scale image features from the input image that are helpful for detailed visual analysis tasks and inject them into the adapter to fuse with the original image features, thereby improving the multimodal large language model's fine-grained visual perception capability of the input image. The following will refer to... Figure 4 and Figure 5This document will provide a detailed description of each step in the data processing method disclosed herein.

[0041] like Figure 3 As shown, firstly, in step S301, the acquired image data can be used to generate a first image feature and a second image feature of the image data through the visual encoder and the visual feature extraction model, respectively, wherein the second image feature is a multi-scale image feature of the image data.

[0042] Optionally, a dual-path visual feature extraction mechanism can be used to perform deep analysis on the input image. The input image data can first undergo normalization preprocessing (e.g., resolution adjustment and channel normalization), and then be fed in parallel into two independent visual processing modules: an image-text aligned visual encoder and a visual feature extraction model, such as... Figure 4 As shown.

[0043] Optionally, the visual encoder for image-text alignment of this disclosure can employ the CLIP model, which, through contrastive learning optimization on large-scale image-text pair datasets, is capable of extracting global image features highly correlated with text semantics. Specifically, the CLIP encoder can map the input image data into a high-dimensional feature vector. This feature focuses on capturing the overall semantic information of the input image data (such as scene category, object attributes, etc.), but due to its architecture design biased towards high-level semantic alignment, its ability to resolve local details (such as edges, textures, etc.) is relatively weak. Therefore, in the embodiments of this disclosure, simultaneously, the input image data is fed into a visual feature extraction model for extracting multi-scale image features.

[0044] Optionally, the visual feature extraction model of this disclosure can employ the SAM-2 (Segment Anything Model v2) model, optimized for multi-scale visual features, as a supplementary vision expert model. This model, based on a hierarchical Transformer architecture and mask decoding mechanism, can extract multi-scale local features from different levels (such as shallow, mid-, and deep layers). By stitching these hierarchical feature maps, the multi-scale features generated by SAM-2 can retain rich detail information, with a spatial resolution significantly higher than CLIP features, enabling precise localization of microstructures (such as cell boundaries in medical images or minute defects in industrial images). Of course, it should be understood that the aforementioned CLIP model and SAM-2 model are used only as examples and not as limitations in this disclosure; other models can also be used to perform the above feature extraction in the data processing method of this disclosure.

[0045] For example, the above dual-path visual feature extraction mechanism can be represented as follows:

[0046] (1)

[0047] Among them, X v This represents the input image data, g represents the image-text alignment visual encoder, and Z represents the input image data. v1 Z represents the first image feature obtained by the image-text alignment visual encoder; f represents the visual feature extraction model; Z v2 This represents the second image feature (multi-scale image feature) obtained by the visual feature extraction model after stitching.

[0048] Based on the first and second image features obtained through step S301 above, the framework proposed in this disclosure can then inject image features containing rich multi-scale image information into the adapter of a pre-trained multimodal large language model through a trainable feature injector.

[0049] Specifically, in step S302, the first image features and the second image features can be input into a feature injector to generate injected features for each adapter layer of the adapter, wherein the feature injector may include the same adapter layer structure as the adapter.

[0050] like Figure 4 As shown, the first image feature Z generated by the image-text aligned visual encoder v1 and the second image feature Z generated by the visual feature extraction model v2 The feature injector can be input in parallel to generate injected features I for each adapter layer of the adapter. v .

[0051] According to embodiments of this disclosure, the feature injector may include a second number of clone adapter layers, and the adapter includes the second number of adapter layers, wherein one clone adapter layer in the feature injector corresponds to one adapter layer in the adapter.

[0052] Optionally, the feature injector can adopt the same hierarchical structure design as the adapter of the pre-trained multimodal large language model and maintain a strict hierarchical correspondence with the adapter. For example, if the original adapter contains N layers (i.e., N adapter layers), the feature injector can also contain N cloned adapter layers, which is exactly the same as the total number of adapter layers in the adapter, forming a one-to-one hierarchical mapping architecture to generate injected features for the corresponding adapter through each cloned adapter layer.

[0053] Optionally, each clone adapter layer can completely replicate the network configuration of the corresponding layer in the adapter in terms of structural parameters, including core parameters such as layer type (e.g., fully connected layer or convolutional layer), number of neurons, and activation function, to ensure isomorphism in feature processing capabilities, enabling each clone adapter layer to generate customized injected features specifically for its corresponding adapter layer.

[0054] Optionally, each adapter layer in the adapter can perform a specific function in cross-modal feature alignment. For example, the first adapter layer can be responsible for receiving the raw image features extracted by the visual encoder, initially adjusting the feature dimensions through linear projection to make them initially compatible with the text embedding space of the large language model. Subsequent adapter layers can gradually decouple redundant information in the visual features and enhance the expression of key features related to text semantics. For example, they can capture cross-modal association patterns through non-linear activation functions (such as GELU), and can also map the processed visual features to dimensions fully compatible with the large language model to generate image tokens that can be directly concatenated to the text token sequence. Each adapter layer retains the original feature information through residual connections to avoid gradient vanishing, while minimizing interference with the stability of the pre-trained model through parameter-efficient design. This hierarchical processing mechanism enables visual features to be gradually adapted to the text semantic space in the adapter, ultimately achieving seamless integration of cross-modal joint inference.

[0055] According to embodiments of this disclosure, inputting the first image feature and the second image feature into a feature injector to generate injected features for each adapter layer of the adapter may include: generating original injected features corresponding to each clone adapter layer in the feature injector based on the first image feature and the weighted second image feature; and for each clone adapter layer in the feature injector, weighting the original injected features corresponding to the clone adapter layer to generate injected features for the adapter layer corresponding to the clone adapter layer.

[0056] Optionally, in the feature injector, the input second image features (multi-scale features) can first undergo dynamic weighting. Considering that the first and second image features typically have different channel dimensions, if element-wise addition (or fusion after concatenation) is required, it must be ensured that their channel dimensions are consistent. Therefore, the core objective of this weighting process is not only to assign importance weights to different feature channels, but also to map the channel dimensions of the second image features to be the same as those of the first image features through a linear transformation.

[0057] For example, this weighting operation is implemented through a learnable attention mechanism, which can be represented as a lightweight fully connected network whose output weight matrix matches the channel dimension of the first image feature to map the second image feature to the feature dimension of the first image feature. The weighted second image feature can then be fused element-wise with the first image feature (global semantic feature) to form an enhanced combined feature.

[0058] Optionally, at each subsequent clone adapter layer, the output of the previous clone adapter layer can be used as its input to generate the original injected features through forward propagation. This generation process strictly follows the computational paradigm of the corresponding adapter layer. For example, if the corresponding adapter layer uses a multi-head attention mechanism, then the clone adapter layer will also perform the same attention computation process.

[0059] Optionally, after generating the original injected features, the original injected features need to be weighted to adaptively adjust the contribution of multi-scale features in different adapter layers. For example, local detail features are strengthened in shallow adapters, while semantic association features are emphasized in deep adapters.

[0060] Through progressive processing, the feature injector can generate corresponding injected features for each network layer of the original adapter, thereby embedding multi-scale visual information into the feature space of the adapter in a parameter-tunable manner.

[0061] According to embodiments of this disclosure, the feature injector may further include a first number of convolutional layers; wherein the first number of convolutional layers includes a second number of convolutional layers, and for each of the second number of clone adapter layers, one of the second number of convolutional layers is associated with the clone adapter layer, the convolutional layer being used to weight the features output by the clone adapter layer.

[0062] Alternatively, the above weighting process can be implemented using the corresponding convolutional layer in the feature injector. Figure 5 This is a schematic diagram illustrating the injection of features into an adapter using a feature injector according to an embodiment of the present disclosure.

[0063] like Figure 5 As shown, taking an adapter with two adapter layers as an example, the visual feature extraction model can have two clone adapter layers based on this, where there is a one-to-one correspondence between each adapter layer and the clone adapter layer.

[0064] According to embodiments of this disclosure, inputting the first image feature and the second image feature into a feature injector to generate injected features for each adapter layer of the adapter may include: processing the second image feature using one of the first number of convolutional layers; generating an original injected feature corresponding to each clone adapter layer in the feature injector based on the first image feature and the processed second image feature; and for each clone adapter layer in the feature injector, processing the original injected feature corresponding to the clone adapter layer using a convolutional layer associated with the clone adapter layer to generate injected features for the adapter layer corresponding to the clone adapter layer.

[0065] Optionally, a convolutional layer can be placed before multiple clone adapter layers (e.g., Figure 5 The convolutional layer 1 in the first image module performs dynamic weighting processing on the input second image features (multi-scale features), where the second image features can be mapped to the feature dimensions of the first image features. The convolutionally processed second image features can be concatenated with the image-text aligned global features (first image features) along the channel dimension to form an enhanced combined feature. Subsequently, this combined feature can be input into the first clone adapter layer of the feature injector (e.g., ...). Figure 5 (Cloning adapter layer 1 in the middle).

[0066] Optionally, the output features of each cloned adapter layer can be further processed through a convolutional layer for channel dimension adjustment and feature scaling to adaptively adjust the contribution of the output feature in the corresponding adapter layer. For example, taking an adapter with two adapter layers as an example, for... Figure 5 The output features of clone adapter layer 1 and clone adapter layer 2 can be processed by convolutional layer 2 and convolutional layer 3 respectively to control the injection of features into the corresponding adapter layer (adapter layer 1 or adapter layer 2) and adjust their contribution to the corresponding adapter layer.

[0067] In step S303, the first image features can be input into the adapter, and injected features for each adapter layer can be injected into the adapter layer to generate an image token corresponding to the image data. The generated image token has a feature dimension that matches the input of the large language model.

[0068] As described above, the feature injector can simultaneously inject the first image feature Z output by the image-text aligned visual encoder. v1 The second image feature Z obtained from the visual expert model v2As input, injected features are generated for each network layer of the adapter. Based on this, the adapter of the pre-trained multimodal large language model can receive image features output from the image-text aligned visual encoder and injected features generated by the feature injector, generating the final image token to be fed into the large language model, such as... Figure 4 As shown. The calculation process can be represented as follows:

[0069] (2)

[0070] Where q represents the feature injector, I v H represents the injected features generated by the feature injector; W represents the adapter of the multimodal large language model; H represents the features generated by the feature injector. v This represents the generated image token.

[0071] According to embodiments of this disclosure, injecting injection features for each adapter layer of the adapter to generate an image token corresponding to the image data may include: for each adapter layer in the second number of adapter layers except the last adapter layer, fusing the features output by the adapter layer and the injection features for the adapter layer, and using the fusion result as the input to the next adapter layer; for the last adapter layer in the second number of adapter layers, fusing the features output by the last adapter layer and the injection features for the last adapter layer, and outputting the fusion result as an image token corresponding to the image data.

[0072] Optionally, for each injected feature generated per adapter layer, the corresponding injected feature can be injected sequentially into each adapter layer. Specifically, in the feature injection of the adapter, multi-scale visual information can be gradually integrated into the semantic space of the pre-trained model through a hierarchical fusion mechanism.

[0073] For example, in a structure containing N adapter layers, each of the first N-1 adapter layers can employ a dynamic fusion strategy. That is, when input features enter the i-th adapter layer, this layer can first perform a predefined feature transformation (e.g., linear projection or multi-head attention computation) to generate an intermediate feature representation. Furthermore, the injected features (containing multi-scale detail information) generated by the feature injector for the current adapter layer can be element-wise weighted and added to these intermediate features using learnable scaling factors (i.e., convolutional layers). This preserves the original semantic processing capabilities of the adapter layer while enhancing the expression of local details through the injected features. Optionally, the fusion result of the i-th adapter layer can be used as the input feature of the next adapter layer (i.e., the i+1-th adapter layer), forming a cascaded information transmission chain.

[0074] Optionally, for the last adapter layer (the Nth adapter layer), after the feature transformation of the adapter layer is completed, the injected features of the last adapter layer are fused with the intermediate features in the same way as the previous N-1 adapter layers. However, the fusion result is no longer passed to subsequent layers, but is directly output as an image token. All injected features are strictly aligned with the output of the corresponding adapter layer through zero-convolution or linear transformation.

[0075] For example, such as Figure 5 As shown, taking an adapter with two adapter layers as an example, the feature injection process can be represented as follows:

[0076] (3)

[0077] Where C1, C2 and C3 respectively represent Figure 5 Convolutional layer 1, convolutional layer 2, and convolutional layer 3 in the middle, I in I1 represents the feature of input clone adapter layer 1 q1, I2 represents the feature of input clone adapter layer 2 q2, and I3 represents the output feature of clone adapter layer 2. W1 and W2 represent adapter layer 1 and adapter layer 2, respectively.

[0078] According to embodiments of this disclosure, the feature injector is trained individually after the visual encoder, the visual feature extraction model, the adapter, and the large language model have been trained, with the parameters of the visual encoder, the visual feature extraction model, the adapter, and the large language model frozen. During the training of the feature injector, all parameters of each convolutional layer are initialized to zero, and each clone adapter layer is initialized with the parameters of the corresponding adapter layer in the adapter. According to embodiments of this disclosure, the feature injector uses the same loss function as used in the pre-training of the multimodal large language model.

[0079] As described above, the data processing method disclosed herein keeps the original pre-trained multimodal large language model unchanged and directly utilizes the weights of the multimodal large language model pre-trained on large-scale data. On this basis, a feature injector can be trained separately to achieve efficient injection of multi-scale information while maintaining the visual-text feature alignment capability of the original model, thereby enhancing the visual perception capability of the multimodal large language model.

[0080] Specifically, for a feature injector with the structure described above, in the initial stage of its training process, the parameters of each clone adapter layer can be directly copied and initialized by its corresponding adapter layer to ensure that the initial state is consistent with the pre-trained parameters.

[0081] Optionally, the aforementioned convolutional layers can be zero convolutional layers (Zero Conv), i.e., networks with convolutional kernel parameters initialized to zero. This avoids interference with the original adapter function in the early stages of feature injection, while allowing the fusion weights of multi-scale features to be learned progressively during training. Based on this design, the coefficients of each clone adapter layer are initialized to zero in the early stages of training and are gradually optimized as the training progresses. This allows the model to be dominated by pre-trained parameters in the initial stage, and then progressively enhance the contribution of injected features. This balances the dual requirements of cross-modal alignment stability and improved detail perception capability, preserving the image-text alignment capability of the pre-trained adapter while achieving targeted enhancement of detail features through the progressive learning mechanism of zero convolution.

[0082] Optionally, the training of the feature injector of this disclosure can follow the principle of parameter freezing and progressive enhancement, that is, keeping all parameters of the pre-trained multimodal large language model and the trained visual feature extraction model frozen (e.g., in...). Figure 4 and Figure 5 The parts highlighted in dark boxes indicate that their parameters are frozen, and all parameters of the convolutional layers are initialized with zeros. All clone adapter layers are initialized with the parameters of the adapter layers of the pre-trained multimodal large language model to ensure safe inheritance of knowledge from the trained model and targeted optimization of visual perception capabilities.

[0083] Specifically, in the initial training phase, all parameters of the pre-trained multimodal large language model (including the visual encoder, adapter, and large language model) are frozen, and the parameters of the visual feature extraction model are also locked to avoid disrupting the existing cross-modal alignment capabilities and visual representations. For the feature injector, its clone adapter layer directly copies the network structure and parameters (such as weight matrices and bias terms) of the corresponding layer in the pre-trained adapter, ensuring functional consistency with the original adapter in the initial state. The convolutional layers are initialized with all zeros for their kernel parameters and bias terms, thereby completely suppressing interference from injected features in the initial training phase.

[0084] Optionally, the training phase of the feature injector can employ a multi-task joint optimization strategy, using image-text pairing data from tasks including but not limited to visual question answering (VQA), optical character recognition (OCR), and image captioning as input. Specifically, after the input training image is processed by the visual encoder and the visual feature extraction model to extract global and multi-scale features respectively, the feature injector generates hierarchical injected features, which are then fused with the adapter's original processing stream.

[0085] Optionally, the loss function during the training phase of the feature injector can completely reuse the objective function of the pre-trained multimodal large language model (such as the cross-entropy loss for text generation tasks, the contrast loss for image-text alignment tasks, etc.), and only update the parameters of the convolutional layers and the clone adapter layers in the feature injector through backpropagation. This training method allows the feature injector to gradually learn how to integrate multi-scale visual features into the pre-trained semantic space in a conflict-minimizing manner during training.

[0086] Through the above processing, multi-scale image features can be progressively injected into the original trained adapter through additional visual feature bypass, so as to enhance the visual perception ability of the multimodal large language model while preserving the original image-text alignment capability.

[0087] In step S304, the image token can be input into the large language model to perform a predetermined image processing task for the image data.

[0088] After obtaining an image token that incorporates multi-scale image information, this token can be input into a large language model (MLM) to perform subsequent downstream image processing tasks. These tasks may include, but are not limited to, VQA, OCR, image caption generation, and document / graph understanding. By utilizing image tokens that incorporate multi-scale image information, the visual perception capabilities of the pre-trained multimodal large language model can be enhanced, thereby improving its performance in these tasks.

[0089] According to embodiments of this disclosure, the data processing method of this disclosure may further include: acquiring language instructions and generating a text token based on the language instructions, wherein the text token has the same feature dimensions as the image token.

[0090] Optionally, for a multimodal large language model, its input may include text data in addition to the image data mentioned above. For example, the text data may be language instructions used to instruct the large language model to perform specific processing on the input image data in order to execute a specific image processing task.

[0091] Alternatively, the acquired text data can be mapped to text tokens to serve as input to a large language model.

[0092] Optionally, the acquired text data can be preprocessed first, including standardization operations (such as converting to lowercase and removing redundant spaces) and special character processing (such as preserving the semantic function of punctuation marks).

[0093] Subsequently, optionally, a pre-trained tokenizer can be used to segment the standardized text data into sub-words to parse the continuous text into a discrete sequence of tokens that the model can understand.

[0094] According to embodiments of this disclosure, inputting the image token into the large language model to perform a predetermined image processing task for the image data may include: inputting the image token and the text token into the large language model to perform an image processing task corresponding to the language instruction.

[0095] Optionally, the multimodal large language model can combine the generated image tokens and text tokens and feed them into the large language model to generate the final text response X in an autoregressive paradigm. a Specifically, image tokens and text tokens can be jointly reasoned using a multimodal large language model (LLM) to ultimately execute image processing tasks that match language instructions.

[0096] Optionally, the image tokens output by the adapter (representing the global semantics and multi-scale detail information of the image) and the text tokens generated by the word segmenter (encoding the semantic content of language instructions) can be input into the large language model through a preset fusion strategy. The large language model can deeply interact with the input sequence through self-attention mechanisms and cross-attention layers, and generate structured responses related to the image processing task at the output layer.

[0097] The above references Figures 3 to 5 The described method, for a trained multimodal large language model, keeps the multimodal large language model unchanged and constructs an additional visual feature bypass for it. This visual feature bypass includes a model that generates multi-scale image features for the input image data and a feature injector with an adapter layer structure identical to the adapter of the multimodal large language model. The generated multi-scale image features are injected into the adapter of the multimodal large language model through this bypass, achieving the fusion of the original image-text alignment features and the multi-scale image features. This allows the image processing task of the large language model to be performed based on the fused image features. This method utilizes an additional visual feature bypass independent of the multimodal large language model to inject multi-scale image features layer by layer into the original image features based on the adapter layer structure. This enhances the visual perception capability of the multimodal large language model while maintaining its original image-text alignment capability.

[0098] Figure 6 This is a schematic diagram illustrating a data processing apparatus 600 according to an embodiment of the present disclosure.

[0099] According to embodiments of this disclosure, the data processing device 600 may include a feature extraction module 601, a feature generation module 602, a feature injection module 603, and a task execution module 604.

[0100] The feature extraction module 601 can be configured to generate a first image feature and a second image feature of the acquired image data using the visual encoder and the visual feature extraction model, respectively, wherein the second image feature is a multi-scale image feature of the image data. Optionally, the feature extraction module 601 can perform the operations described above with reference to step S301.

[0101] Optionally, a dual-path visual feature extraction mechanism can be used to perform deep analysis on the input image. The input image data can first undergo standardization preprocessing, and then be input in parallel to two independent visual processing modules: an image-text aligned visual encoder and a visual feature extraction model.

[0102] Based on the first and second image features obtained through the feature extraction module 601, the image features containing rich multi-scale image information can then be injected into the adapter of the pre-trained multimodal large language model through a trainable feature injector.

[0103] The feature generation module 602 can be configured to input the first image features and the second image features into a feature injector to generate injected features for each adapter layer of the adapter, wherein the feature injector includes the same adapter layer structure as the adapter. Optionally, the feature generation module 602 can perform the operations described above with reference to step S302.

[0104] Optionally, the feature injector can adopt the same hierarchical structure design as the adapter of the pre-trained multimodal large language model and maintain a strict hierarchical correspondence with the adapter. For example, if the original adapter contains N layers (i.e., N adapter layers), the feature injector can also contain N cloned adapter layers, which is exactly the same as the total number of adapter layers in the adapter, forming a one-to-one hierarchical mapping architecture to generate injected features for the corresponding adapter through each cloned adapter layer.

[0105] Optionally, each clone adapter layer can completely replicate the network configuration of the corresponding layer in the adapter in terms of structural parameters, including core parameters such as layer type (e.g., fully connected layer or convolutional layer), number of neurons, and activation function, to ensure isomorphism in feature processing capabilities, enabling each clone adapter layer to generate customized injected features specifically for its corresponding adapter layer.

[0106] Optionally, in the feature injector, the input second image features (multi-scale features) can first undergo dynamic weighting. Considering that the first and second image features typically have different channel dimensions, if element-wise addition (or fusion after concatenation) is required, it must be ensured that their channel dimensions are consistent. Therefore, the core objective of this weighting process is not only to assign importance weights to different feature channels, but also to map the channel dimensions of the second image features to be the same as those of the first image features through a linear transformation.

[0107] Optionally, at each subsequent clone adapter layer, the output of the previous clone adapter layer can be used as its input to generate the original injected features through forward propagation. This generation process strictly follows the computational paradigm of the corresponding adapter layer. For example, if the corresponding adapter layer uses a multi-head attention mechanism, then the clone adapter layer will also perform the same attention computation process.

[0108] Optionally, after generating the original injected features, the original injected features need to be weighted to adaptively adjust the contribution of multi-scale features in different adapter layers. For example, local detail features are strengthened in shallow adapters, while semantic association features are emphasized in deep adapters.

[0109] Through progressive processing, the feature injector can generate corresponding injected features for each network layer of the original adapter, thereby embedding multi-scale visual information into the feature space of the adapter in a parameter-tunable manner.

[0110] Optionally, the aforementioned weighting process can be implemented using a corresponding convolutional layer in the feature injector. For example, a convolutional layer can be placed before multiple clone adapter layers to perform dynamic weighting of the input second image features (multi-scale features), where the second image features can be mapped to the feature dimensions of the first image features. The convolutionally processed second image features can then be concatenated with the image-text aligned global features (first image features) along the channel dimension to form an enhanced combined feature. This combined feature can then be input into the first clone adapter layer of the feature injector.

[0111] Optionally, the output features of each clone adapter layer can be further processed through a convolutional layer for channel dimension adjustment and feature scaling to adaptively adjust the contribution of the output feature in the corresponding adapter layer.

[0112] The feature injection module 603 can be configured to input the first image features into the adapter and inject features for each adapter layer of the adapter to generate an image token corresponding to the image data, the generated image token having a feature dimension that matches the input of the large language model. Optionally, the feature injection module 603 can perform the operations described above with reference to step S303.

[0113] Optionally, in the feature injection module 603, the adapter of the pre-trained multimodal large language model can receive image features output by the image-text aligned visual encoder and injected features generated by the feature injector to generate the final image token sent into the large language model.

[0114] Optionally, for each injected feature generated per adapter layer, the corresponding injected feature can be injected sequentially into each adapter layer. Specifically, in the feature injection of the adapter, multi-scale visual information can be gradually integrated into the semantic space of the pre-trained model through a hierarchical fusion mechanism.

[0115] Optionally, the aforementioned convolutional layers can be zero-convolutional layers, i.e., networks with convolutional kernel parameters initialized to zero. This avoids interference with the original adapter function in the early stages of feature injection, while allowing the fusion weights of multi-scale features to be learned progressively during training. Based on this design, the coefficients of each clone adapter layer are initialized to zero in the early stages of training and are gradually optimized as the training progresses. This allows the model to be dominated by pre-trained parameters in the initial stage, and then progressively enhance the contribution of injected features. This balances the dual requirements of cross-modal alignment stability and improved detail perception capabilities, preserving the image-text alignment capability of the pre-trained adapter while achieving targeted enhancement of detail features through the progressive learning mechanism of zero-convolution.

[0116] Through the processing of the feature extraction module 601, the feature generation module 602, and the feature injection module 603, multi-scale image features can be progressively injected into the original trained adapter through additional visual feature bypass, so as to enhance the visual perception capability of the multimodal large language model while retaining the original image-text alignment capability of the multimodal large language model.

[0117] The task execution module 604 can be configured to input the image token into the large language model to perform a predetermined image processing task for the image data. Optionally, the task execution module 604 can perform the operations described above with reference to step S304.

[0118] After obtaining the image token that integrates multi-scale image information, the task execution module 604 can input the image token into the large language model for subsequent downstream image processing tasks. For example, downstream image processing tasks may include, but are not limited to, VQA, OCR, image caption generation, and document / chart understanding. By utilizing the image token that integrates multi-scale image information, the visual perception capability of the pre-trained multimodal large language model can be enhanced, thereby improving the performance of the multimodal large language model in these tasks.

[0119] According to another aspect of this disclosure, a data processing apparatus is also provided. Figure 7 A schematic diagram of a data processing apparatus 2000 according to an embodiment of the present disclosure is shown.

[0120] like Figure 7 As shown, the data processing device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code, which, when executed by the one or more processors 2010, can perform the data processing method described above.

[0121] The processor in the embodiments of this disclosure can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0122] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0123] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 8 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 8As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication in the data processing method provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 8 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 8 One or more components in the computing device shown.

[0124] According to another aspect of this disclosure, a computer-readable storage medium is also provided. The computer storage medium stores computer-readable instructions. When the computer-readable instructions are executed by a processor, a data processing method according to embodiments of this disclosure described with reference to the above-drawn figures can be performed. The computer-readable storage medium in the embodiments of this disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0125] Embodiments of this disclosure also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a data processing method according to embodiments of this disclosure.

[0126] Embodiments of this disclosure provide a data processing method, apparatus, device, computer program product, and computer-readable storage medium.

[0127] The method provided in the embodiments of this disclosure maintains the trained multimodal large language model unchanged by constructing an additional visual feature bypass for it. This visual feature bypass includes a model that generates multi-scale image features from the input image data and a feature injector with an adapter layer structure identical to the adapter of the multimodal large language model. The generated multi-scale image features are injected into the adapter of the multimodal large language model through this bypass, achieving the fusion of the original image-text alignment features and the multi-scale image features. This allows the image processing task of the large language model to be performed based on the fused image features. The method of the embodiments of this disclosure utilizes an additional visual feature bypass independent of the multimodal large language model to inject multi-scale image features layer by layer into the original image features based on the adapter layer structure. This enhances the visual perception capability of the multimodal large language model while maintaining its original image-text alignment capability.

[0128] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0130] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A data processing method applied to a pre-trained multimodal large language model, wherein the pre-trained multimodal large language model includes a visual encoder for image-text alignment, an adapter, and a large language model, the method comprising: For the acquired image data, a first image feature and a second image feature of the image data are generated by the visual encoder and the visual feature extraction model, respectively, wherein the second image feature is a multi-scale image feature of the image data; The first image feature and the second image feature are input into a feature injector to generate injected features for each adapter layer of the adapter, wherein the feature injector includes the same adapter layer structure as the adapter; The first image features are input into the adapter, and injected features for each adapter layer are injected into the adapter to generate an image token corresponding to the image data. The generated image token has a feature dimension that matches the input of the large language model. The image token is input into the large language model to perform a predetermined image processing task for the image data.

2. The method as described in claim 1, wherein, The feature injector includes a first number of convolutional layers and a second number of clone adapter layers, and the adapter includes the second number of adapter layers; Wherein, one clone adapter layer in the feature injector corresponds to one adapter layer in the adapter; Wherein, the first number of convolutional layers includes the second number of convolutional layers, and for each of the second number of clone adapter layers, one of the second number of convolutional layers is associated with the clone adapter layer, the convolutional layer being used to weight the features output by the clone adapter layer.

3. The method as described in claim 2, wherein, The step of inputting the first image features and the second image features into the feature injector to generate injected features for each adapter layer of the adapter includes: The second image features are processed using one of the first number of convolutional layers; Based on the first image features and the processed second image features, original injected features corresponding to each clone adapter layer are generated through each clone adapter layer in the feature injector; For each clone adapter layer in the feature injector, the original injected features corresponding to the clone adapter layer are processed using a convolutional layer associated with the clone adapter layer to generate injected features for the adapter layer corresponding to the clone adapter layer.

4. The method of claim 2, wherein, The feature injector is trained separately after the visual encoder, the visual feature extraction model, the adapter, and the large language model have been trained, with the parameters of the visual encoder, the visual feature extraction model, the adapter, and the large language model frozen. In the training of the feature injector, all parameters of each convolutional layer are initialized to zero, and each clone adapter layer is initialized with the parameters of the corresponding adapter layer in the adapter.

5. The method of claim 4, wherein, In the training of the feature injector, the same loss function used in the pre-training of the multimodal large language model is employed.

6. The method of claim 1, wherein, The feature injector includes a second number of clone adapter layers, and the adapter includes the second number of adapter layers, wherein one clone adapter layer in the feature injector corresponds to one adapter layer in the adapter; The step of inputting the first image features and the second image features into the feature injector to generate injected features for each adapter layer of the adapter includes: Based on the first image features and the weighted second image features, the original injected features corresponding to each clone adapter layer are generated through each clone adapter layer in the feature injector. For each clone adapter layer in the feature injector, the original injected features corresponding to the clone adapter layer are weighted to generate injected features for the adapter layer corresponding to the clone adapter layer.

7. The method as described in claim 3 or 6, wherein, The step of injecting injection features for each adapter layer of the adapter to generate an image token corresponding to the image data includes: For each adapter layer in the second number of adapter layers except the last adapter layer, the features output by the adapter layer and the injected features for the adapter layer are fused, and the fusion result is used as the input of the next adapter layer; For the last adapter layer in the second number of adapter layers, the features output by the last adapter layer and the injected features for the last adapter layer are fused, and the fusion result is output as an image token corresponding to the image data.

8. The method of claim 1, further comprising: Obtain language instructions and generate a text token based on the language instructions, the text token having the same feature dimensions as the image token; The step of inputting the image token into the large language model to perform a predetermined image processing task for the image data includes: The image token and the text token are input into the large language model to execute the image processing task corresponding to the language instruction.

9. A data processing apparatus applied to a pre-trained multimodal large language model, the pre-trained multimodal large language model comprising a visual encoder for image-text alignment, an adapter, and a large language model, the apparatus comprising: The feature extraction module is configured to generate a first image feature and a second image feature of the acquired image data through the visual encoder and the visual feature extraction model, respectively, wherein the second image feature is a multi-scale image feature of the image data; A feature generation module is configured to input the first image feature and the second image feature into a feature injector to generate injected features for each adapter layer of the adapter, wherein the feature injector includes the same adapter layer structure as the adapter; A feature injection module is configured to input the first image features into the adapter and inject injected features for each adapter layer of the adapter to generate an image token corresponding to the image data, the generated image token having a feature dimension matching the input of the large language model; and The task execution module is configured to input the image token into the large language model to perform a predetermined image processing task for the image data.

10. A data processing device, comprising: One or more processors; as well as One or more memories storing a computer-executable program, which, when executed by the processor, performs the method of any one of claims 1-8.

11. A computer program product stored on a computer-readable storage medium and comprising computer instructions that, when executed by a processor, cause a computer device to perform the method of any one of claims 1-8.

12. A computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

Images