A multimodal model visual perception capability enhancement method, device and medium

Abstract

The application discloses a multimodal model visual perception ability enhancement method and device and medium, wherein the method comprises the following steps: collecting image data, and proposing a text question according to the image; selecting a plurality of multimodal language large models trained based on the same language model, and merging the parameters of the language models; based on the input image, using a plurality of visual encoders to extract visual features; inputting the text, and extracting features of the input text through the language model to obtain text features; splicing the visual features and the text features to obtain fusion features; inputting the fusion features into the language model for reasoning, and outputting a reasoning result. The application does not require additional training, can effectively improve the performance of a multimodal task, and reduces deployment overhead. In addition, by using the technology of the application, the advantages of different visual encoders can be fused, the visual perception effect of a multimodal large model is significantly improved, and the demand for computing resources is reduced. The application can be widely applied in the field of artificial intelligence.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a multi-modal model visual perception ability enhancement method, device and medium. BACKGROUND

[0002] With the wide application of multi-modal large models (MLLMs) in visual-linguistic tasks, how to improve the visual perception ability of the model has become an important research direction. Existing technical solutions mostly enhance the visual performance of the model by designing a more powerful visual encoder, but these solutions usually need to align the visual encoder with the language model and perform a large amount of multi-modal training, resulting in expensive computing overhead and complex training process. For example, aligning a visual encoder with a 7B parameter order language model usually requires thousands of hours of GPU training time, which is extremely costly. In addition, even after fine-tuning the model, a single model still has limitations in visual perception when dealing with complex visual tasks, making it difficult to fully capture diverse visual information.

[0003] Research has found that different multi-modal models often have their own advantages and disadvantages in visual perception. For example, one model may be better at recognizing local details in an image, while another model may be better at capturing global information. This difference in visual perception ability indicates that a single model may not provide the best performance when faced with complex and diverse visual tasks, but integrating the perception abilities of multiple models can make up for the shortcomings of a single model.

[0004] Existing solutions, such as ensemble learning, combine the outputs of multiple models to take advantage of different models, but this method usually requires running multiple complete models simultaneously, resulting in a waste of a large amount of memory and computing resources. Although some multi-modal large models attempt to reduce computing overhead by splicing features from different visual encoders, they still need to perform expensive alignment training on the language model and visual encoder, limiting their feasibility in practical applications.

[0005] Therefore, there is an urgent need for a new method that can effectively utilize the visual encoders of multiple pre-trained models to improve the visual perception ability and overall performance of the model in multi-modal tasks without increasing the training cost. SUMMARY

[0006] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a training-free multi-modal model visual perception ability enhancement method, device and medium

[0007] The first technical solution adopted by the present application is:

[0008] A multimodal model visual perception ability enhancement method, comprising the following steps:

[0009] Collect image data and propose text questions according to the images;

[0010] Select multiple multimodal language large models trained based on the same language model, and combine the parameters of the language model;

[0011] Based on the input image, use multiple visual encoders to extract rich visual features;

[0012] Input text, extract features of the input text through the language model, and obtain text features;

[0013] Splice the visual features and the text features to obtain fusion features;

[0014] Input the fusion features into the language model for inference, and output the inference result.

[0015] Further, the image data is collected, and the text questions are proposed according to the images, comprising:

[0016] Use a high-resolution photographic device to collect image data;

[0017] After collecting the images, text questions are proposed for each image, which will be used for inference of multimodal tasks.

[0018] Further, the multiple multimodal language large models trained based on the same language model are selected, comprising:

[0019] When selecting a multimodal large model, the following three key modules need to be considered: pre-processing module, modal-specific visual encoder and modal-specific mapping module;

[0020] The function of the pre-processing module is to standardize and preliminarily convert the input image data, so that the subsequent visual encoder can effectively process the image information; the visual encoder is responsible for converting the pre-processed image data into visual features; the mapping module is responsible for converting the visual features extracted by the visual encoder into an embedding format that can be processed by the language model.

[0021] Further, the step of selecting multiple multimodal language large models trained based on the same language model further comprises:

[0022] After the visual features and the text features are combined after pre-processing, encoding and mapping, they are input into the shared language model for inference; wherein the parameters of the language model are shared in all multimodal models, ensuring the consistency of the language part;

[0023] The language model generates the final inference result by processing the text and visual information;

[0024] In selecting the multi-modal large models, it is necessary to ensure that the language model parts of these multi-modal large models are based on the same pre-training model (parameters denoted as Θ), i.e., the language model parameters of each model remain consistent.

[0025] Further, the parameters of the merged language model include:

[0026] After selecting the multi-modal large models, the language model parameters from different multi-modal large models are merged to generate a unified language model capable of interpreting visual information from multiple visual encoders; the specific formula is represented as:

[0027]

[0028] In the formula, Θ pre represents the parameters of the pre-trained language model, Θ i represents the language model parameters of the i-th multi-modal large model, M is the number of multi-modal large models, and λ is an adjustable merging coefficient; (Θ i -Θ pre ) is the parameter of each model relative to its pre-trained language model.

[0029] Further, the extraction of visual features based on the input image using multiple visual encoders includes:

[0030] Based on the input image x i , multiple different visual encoders are used for feature extraction, and each visual encoder focuses on extracting different levels of features of the image;

[0031] The extracted visual features V i are used in conjunction with the language model as one of the inputs in the multi-modal inference process.

[0032] Further, the splicing of visual features and text features includes:

[0033] The multiple visual features are spliced as:

[0034] V F = [V1, V2, …, V i ]

[0035] In the formula, V F represents the spliced feature vector, and [] represents splicing multiple visual features V i along a certain dimension.

[0036] Further, the inference process of the language model is represented as:

[0037] y = f (V F , T; Θ)

[0038] wherein V F is the concatenated feature vector, T is the text feature, and Θ is the parameter of the language model.

[0039] The second technical solution adopted by the present application is:

[0040] An electronic device, comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the multi-modal model visual perception capability enhancement method as described above.

[0041] The fourth technical solution adopted by the present application is:

[0042] A computer-readable storage medium, the storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement the multi-modal model visual perception capability enhancement method as described above.

[0043] The fifth technical solution adopted by the present application is:

[0044] A computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to make the computer device execute the multi-modal model visual perception capability enhancement method as described above.

[0045] The beneficial effects of the present application are: the present application can effectively improve the performance of multi-modal tasks without additional training, and reduces the deployment overhead. In addition, by using the technology of the present application, the advantages of different visual encoders can be fused to significantly improve the visual perception effect of the multi-modal large model, while reducing the computing resource demand. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0047] Figure 1is a step flow chart of a multi-modal model visual perception ability enhancement method in an embodiment of the present application;

[0048] Figure 2 is a schematic diagram of a multi-modal model visual perception ability enhancement method in an embodiment of the present application. DETAILED DESCRIPTION

[0049] Embodiments of the present application are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0050] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0051] In the description of the present application, the meaning of several is one or more, the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, above, below, etc. are understood as including the number. If the first, second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of technical features indicated.

[0052] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0053] With the wide application of multi-modal large models in the field of artificial intelligence, improving the visual perception ability of the model has become an important research direction. Existing multi-modal models usually rely on the combination of a single visual encoder and a language model. Although this method can complete multi-modal tasks to a certain extent, in complex scenes or diversified visual inputs, a single encoder often has difficulty comprehensively capturing and analyzing the rich information in the image. In addition, the existing technology needs to realign the visual encoder and the language model and perform a large amount of training, which is costly and difficult to effectively respond to multiple visual perception needs. To overcome these limitations, the present application proposes a training-free multi-modal model visual perception ability enhancement scheme, which can significantly improve the visual perception ability of the model in multi-modal tasks without increasing additional training costs. By combining multiple visual encoders based on the same language model fine-tuning and merging the parameters of the language model, the present application scheme realizes the excellent performance of the multi-modal large model in complex tasks.

[0054] Embodiment 1

[0055] As shown in Figure 1 and Figure 2 , the present embodiment provides a training-free multi-modal model visual perception ability enhancement method, which specifically includes the following steps:

[0056] S1, collect image data and propose text questions according to the images.

[0057] As an optional implementation, step S1 specifically includes the following steps:

[0058] S1-1: First, use a high-resolution photography device to collect image data. The photography device can be a digital camera, a smartphone, or a professional photography system. The user should ensure that the device is in proper working condition and avoid image blur or distortion during the shooting process. All collected images will be used for subsequent multi-modal model inference tasks.

[0059] S1-2: When collecting images, the user can take pictures in different scenes according to needs, such as indoors, outdoors, natural light, or artificial lighting, etc. The collected images can include any content of interest, such as still life, people, scenery, etc. The collection process does not need to consider the diversity or complexity of the images, the focus is on obtaining high-quality image data for subsequent testing and analysis for multi-modal tasks.

[0060] S1-3: After collecting the images, the user needs to propose text questions for each image, which will be used for inference in multi-modal tasks. The questions can be freely set according to the image content, ensuring that the questions can be related to the visual content of the image. For example:

[0061] a) If the image contains vehicles, the user can ask: "How many vehicles are in the image?"

[0062] b) If the image contains vehicles, the user can ask: "How many vehicles are in the image?"

[0063] These questions will be inputted as the model's input along with the image data for subsequent multimodal reasoning.

[0064] S2, select multiple multimodal large language models trained based on the same language model, and merge the parameters of the language model.

[0065] In one embodiment, step S2 specifically includes the following steps:

[0066] S2-1: When selecting a multimodal large language model (MLLM), the following three key modules need to be considered for their composition and function: preprocessing module, modality-specific visual encoder, and modality-specific mapping module. These modules work together to effectively integrate input visual data with the language model, enabling cross-modal reasoning.

[0067] a) Preprocessing module

[0068] The main function of the preprocessing module is to standardize and preliminarily convert the input image data, so that the subsequent encoder can effectively process the image information. Specific operations may include image segmentation processing (such as dividing the image into local regions) or some basic preprocessing operations (such as normalization, scaling, etc.). The preprocessing module ensures that the input image is standardized so that the encoder can efficiently extract meaningful visual features from it.

[0069] Mathematically expressed as: x′ i = Preprocessing i (x i ); where x i is the input original image data, x′ i is the preprocessed data, Preprocessing i represents the preprocessing module of the i-th model.

[0070] b) Visual encoder

[0071] The visual encoder is responsible for converting the preprocessed image data into visual features. Different multimodal models may use different encoder architectures, including convolutional neural networks (CNN) or Transformer-based architectures (such as Vision Transformer, ViT). These visual encoders extract high-dimensional features from images, which represent important information in the image for subsequent cross-modal reasoning.

[0072] The output of the visual encoder is visual feature V, denoted as: V = Env i (x′ i ) where Env i is the visual encoder of the i-th model, x′ i is the pre-processed input image, and V is the visual feature extracted by the encoder.

[0073] c) Mapper

[0074] The mapping module is responsible for converting the visual features V extracted by the encoder into an embedding format that can be processed by the language model. Since the visual features and the input of the language model usually reside in different spaces, the mapping module needs to transform these features so that they can be combined with the text input of the language model for cross-modal reasoning.

[0075] The function of the mapping module is denoted as: V i = Proj i (V) where Proj i is the mapping module of the i-th model, V i is the feature extracted from the visual encoder, and V i is the mapped embedding vector used for combination with the language model.

[0076] d) Language Model

[0077] After pre-processing, encoding, and mapping, the visual features are combined with the text features and input into the shared language model for reasoning. The parameters of the language model are shared across all multi-modal models, ensuring consistency in the language part. The language model generates the final reasoning result by processing the text and visual information. When selecting multi-modal large models, it is necessary to ensure that the language model part of these models is based on the same pre-training model (parameters denoted as Θ), i.e., the language model parameters of each model remain consistent. The visual encoder and mapping module can be designed differently according to task requirements.

[0078] e) Diversified Model Selection

[0079] When selecting models, multi-modal models with different visual encoders and mapping modules are preferred. Different encoders may focus on different aspects when processing visual data, with some being better at extracting local details and others being better at extracting global information. By integrating multiple models with different characteristics, the processing capability for complex visual tasks can be improved, and the performance of the model in multi-modal tasks can be enhanced.

[0080] S2-2: A single language model cannot directly align with multiple encoders from different multimodal large models (MLLMs) because these models are not trained for alignment. Directly training multiple visual encoders for alignment usually requires extremely high computational overhead. Therefore, based on existing research methods, only by merging language model parameters from different visual encoders, the language model can help interpret visual information in multiple visual encoders. This merging process does not require retraining and can effectively reduce computational cost.

[0081] After selecting multiple multimodal large models, we merge language model parameters from different models to generate a unified language model that can interpret visual information from multiple visual encoders. The specific merging process can be represented as:

[0082]

[0083] where Θ pre represents the parameters of the pre-trained language model, Θ i represents the language model parameters of the i-th multimodal large model, M is the number of multimodal large models, and λ is the adjustable merging coefficient. (Θ i - Θ pre ) is the deviation of each model relative to its pre-trained language model.

[0084] S3, based on the input image, use multiple visual encoders to extract rich visual features.

[0085] Exemplarily, in this step, the system extracts visual features of the input image through the visual encoder. Each visual encoder receives the input image data x i and converts it into the corresponding visual feature V i . The specific processing process is as follows:

[0086] First, the input image x i is passed to the visual encoder, which is responsible for analyzing and extracting features from the image. The functions of the visual encoder include identifying local features, global structures, and other key information in the image. The extracted visual features are represented as:

[0087] V i = Proj i (Enc i (Preprocessing i (x i )))

[0088] where V i represents the visual features extracted from the image data x iExtracted visual features. These visual features can contain information about the color, texture, shape, and spatial relationships of the image.

[0089] To improve the model's understanding of complex images, multiple different visual encoders can be used, each focusing on extracting different levels of features from the image. For example, some encoders might be better at capturing local detail information in the image, while others might focus more on global features of the image. By working together in this multi-encoder collaborative manner, the system is able to generate a richer visual feature representation.

[0090] Extracted visual features V i Subsequently used in conjunction with the language model in subsequent steps as one of the inputs in the multimodal inference process.

[0091] S4, input text, extract text features from the input text through the language model.

[0092] In some embodiments, the input text t is processed by the word embedding module of the language model to extract text features. Specifically, the input text is converted into a high-dimensional vector representation through the embedding function T = Embedding(t). This vector contains the semantic information of the text.

[0093] S5, concatenate visual features and text features to obtain fused features.

[0094] The extracted visual features V i are concatenated with the text features T to form multimodal input features for subsequent inference processes. The purpose of the concatenation operation is to combine visual information with text information so that the language model can process information from both modalities simultaneously for multimodal inference.

[0095] First, the system receives the visual features V i extracted from the visual encoder. The concatenation operation can be represented as:

[0096] V F = [V1, V2, …, Vi]

[0097] where V F represents the concatenated multimodal feature vector, and [] represents the concatenation of visual features V i along a certain dimension. In this way, visual information and text information are jointly expressed in the same input space, ensuring that the language model can comprehensively process input information from both modalities. The concatenated joint feature vector preserves the information of the visual and text modalities and combines them into a complete input representation through the concatenation operation. This concatenation operation not only preserves the independence of each modality feature, but also provides richer cross-modal information input for subsequent inference.

[0098] S6, input the fusion features into a language model to perform reasoning and output a reasoning result.

[0099] The spliced multi-modal feature vector is input into the language model for reasoning calculation. The language model generates the final cross-modal reasoning result by processing the joint input containing visual features and text features. The purpose of this step is to utilize the semantic understanding ability of the language model to conduct comprehensive reasoning in combination with visual features.

[0100] The language model utilizes its pre-trained parameters Θ merged Process the input to jointly interpret semantic and visual features. The language model processes the multi-modal information of the input through its internal multi-layer structure in sequence and generates a reasoning result.

[0101] The reasoning process of the language model can be represented as:

[0102] y = f(V F ,T; Θ)

[0103] Where y is the output result of the language model.

[0104] In summary, existing multi-modal large models usually only rely on the alignment of a single visual encoder and a language model to complete multi-modal tasks, resulting in the visual perception ability of the model being limited to the characteristics of the encoder. This method is prone to insufficient or biased information capture when dealing with complex and diverse visual scenes, and cannot fully utilize the multi-dimensional understanding of image features by different visual encoders. In contrast, the present application innovatively proposes a scheme of integrating multiple visual modules, which enables the model to simultaneously obtain multi-level and multi-perspective visual information by splicing and fusing features from different visual encoders. In addition, the present application merges the language model parameters of multiple multi-modal large models without the need for additional retraining of the visual encoder and the language model. This parameter merging method greatly reduces the computational overhead of re-alignment and training, and ensures the reasoning efficiency of the model, enabling the model to maintain high accuracy while significantly reducing computational and time costs.

[0105] By combining multiple visual modules, the present application can significantly improve visual perception ability in multi-modal tasks, especially in tasks that require detailed image analysis or complex scene perception. The model can capture local and global features of the image from multiple angles, making the inference result more accurate and comprehensive. Secondly, the prior art usually requires complex and time-consuming retraining of the model, while the present application can enhance the model without additional training through the parameter merging method, which greatly reduces the difficulty of model development and deployment, and also reduces the required computing resources and costs. Thirdly, the deployment cost of the present application is lower, and the merged model can efficiently process features from multiple visual encoders, and can also run smoothly in resource-limited environments such as embedded devices or edge computing scenarios. Finally, the present application has wide application adaptability, not only suitable for enhancing the visual perception ability of multi-modal large models, but also can play a role in other multi-modal tasks such as text generation, visual question answering, image recognition, etc., significantly improving the accuracy and performance of these tasks. Therefore, the present application scheme has high technical advantages and wide practical application value.

[0106] Embodiment 2

[0107] The embodiments of the present application also provide an electronic device, which includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a multi-modal model visual perception ability enhancement method as shown in Figure 1

[0108] It can be understood that the memory can include a random access memory (RAM) and a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing each of the above methods, etc.; and the data storage area can store data created according to the use of the server, etc.

[0109] ​The processor can include one or more processing cores. The processor connects various parts within the entire server by various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Alternatively, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor can be integrated with one or several combinations of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but be realized by a single chip.

[0110] Since the electronic device is an electronic device corresponding to the multi-modal model visual perception ability enhancement method of the embodiment of the application, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0111] Embodiment 3

[0112] The embodiment of the application also provides a computer readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set are stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to realize a multi-modal model visual perception ability enhancement method as shown in Figure 1

[0113] ​Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0114] Since the storage medium is a storage medium corresponding to the method of enhancing the visual perception ability of a multi-modal model according to the embodiments of the present application, and the principle of solving the problem is similar to the method, the implementation of the storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0115] Embodiment 4

[0116] In some possible implementation manners, various aspects of the method according to the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the method of enhancing the visual perception ability of a multi-modal model according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. The executable computer program codes or "codes" for executing various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0117] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so on.

[0118] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0119] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A method for enhancing visual perception capability of a multi-modal model, characterized in that, The method comprises the following steps: Collecting image data and proposing a text question based on the image; Selecting multiple multimodal large models trained based on the same language model and merging the parameters of the language model; Extracting visual features based on the input image using multiple visual encoders; Inputting the text and extracting features of the input text through the language model to obtain text features; Splicing the visual features and the text features to obtain fusion features; Inputting the fusion features into the multimodal language model for reasoning to output a reasoning result; The collecting image data and proposing a text question based on the image comprises: Collecting image data using a high-resolution photographic device; After collecting the image, a text question is proposed for each image, which will be used for multimodal task reasoning; The selecting multiple multimodal large models trained based on the same language model comprises: When selecting multimodal large models, the following three key modules need to be considered: preprocessing module, visual encoder and mapping module; The function of the preprocessing module is to standardize and preliminarily convert the input image data so that the visual encoder can effectively process the image information; the visual encoder is responsible for converting the preprocessed image data into visual features; The mapping module is responsible for converting the visual features extracted by the visual encoder into an embedding format that can be processed by the language model; After the visual features and the text features are combined after preprocessing, encoding and mapping, they are input into the shared language model for reasoning; the parameters of the language model are shared in all multimodal large models to ensure the consistency of the language part; The multimodal large model generates the final reasoning result by processing the text features and the visual features; When selecting multimodal large models, it is necessary to ensure that the language model part of these multimodal large models is based on the same pre-training model, i.e., the language model parameters of each multimodal large model remain consistent; The merging of the parameters of the language model comprises: After selecting multiple multimodal large models, the language model parameters from different multimodal large models are merged to generate a unified language model that can interpret visual features from multiple visual encoders; the specific formula is as follows: In the formula, Θ pre represents the parameters of the pre-trained language model, Θ i represents the language model parameters of the i-th multi-modal large model, M is the number of multi-modal large models, and λ is an adjustable merging coefficient; (Θ i -Θ pre ) is the difference between the parameters of the language model of each multi-modal large model and the parameters of the pre-trained language model.

2. The method of claim 1, wherein, The extracting visual features based on the input image using multiple visual encoders comprises: Based on the input image x i feature extraction using a plurality of different visual encoders, each visual encoder focusing on extracting different levels of features of the image; The extracted visual features V i For combination with the language model as one of the inputs in the multimodal inference process.

3. The method of claim 1, wherein, The splicing visual features and text features comprises: Splicing multiple visual features: V F = [V1, V2,..., V i ] In the formula, V F The concatenated feature vector is represented by V i Concatenation is performed along a certain dimension.

4. The method of claim 1, wherein, The reasoning process of the language model is represented as follows: y = f(V F , T; Θ) In the formula, V F is the stitched feature vector, T is the text feature, and Θ is the parameter of the language model.

5. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method of any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method of any one of claims 1 to 4.

Images