Multi-modal model visual perception ability enhancement method and device, and medium

Abstract

The invention discloses a multi-modal model visual perception ability enhancement method and device and a medium, and the method comprises the steps: collecting image data, and proposing a text problem according to an image; selecting a plurality of multi-modal language large models trained based on the same language model, and merging parameters of the language models; based on the input image, utilizing a plurality of visual encoders to extract visual features; inputting a text, and performing feature extraction on the input text through the language model to obtain text features; splicing the visual features and the text features to obtain fusion features; and inputting the fusion features into a language model for reasoning, and outputting a reasoning result. According to the method, extra training is not needed, the performance of the multi-modal task can be effectively improved, and the deployment overhead is reduced. Besides, by means of the technology, the visual perception effect of the multi-modal large model can be remarkably improved by fusing the advantages of different visual encoders, and meanwhile the requirement for computing resources is lowered. The method can be widely applied to the field of artificial intelligence.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, device and medium for enhancing the visual perception ability of a multimodal model. Background Art

[0002] With the widespread application of multimodal large models (MLLMs) in visual-language tasks, how to improve the visual perception ability of the model has become an important research direction. Most existing technical solutions enhance the visual performance of the model by designing more powerful visual encoders, but these solutions usually require realigning the visual encoder with the language model and performing a large amount of multimodal training, resulting in expensive computational overhead and complex training process. For example, aligning a visual encoder with a language model with 7B parameters usually requires thousands of hours of GPU training time, which is extremely costly. In addition, even after fine-tuning the model, a single model still has limitations in perception ability when processing complex visual tasks, and it is difficult to fully capture diverse visual information.

[0003] Research has found that different multimodal models often have their own advantages and disadvantages in visual perception. For example, one model may be better at identifying local details in an image, while another model may be better at capturing global information. This difference in visual perception ability indicates that a single model may not provide the best performance when faced with complex and diverse visual tasks, but the shortcomings of a single model can be compensated by integrating the perception capabilities of multiple models.

[0004] Existing solutions, such as ensemble learning, leverage the strengths of different models by simply combining their outputs, but this approach usually requires running multiple complete models simultaneously, resulting in a large waste of memory and computing resources. Although some large multimodal models attempt to reduce computational overhead by concatenating features from different visual encoders, they still require expensive alignment training of the language model and visual encoder, limiting their feasibility in practical applications.

[0005] Therefore, there is an urgent need for a new method that can effectively utilize the visual encoders of multiple pre-trained models without increasing the training cost, so as to improve the visual perception ability and overall performance of the model in multimodal tasks. Summary of the invention

[0006] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a method, device and medium for enhancing the visual perception ability of a multimodal model without training.

[0007] The first technical solution adopted by the present invention is:

[0008] A method for enhancing the visual perception ability of a multimodal model comprises the following steps:

[0009] Collect image data and ask text questions based on the image;

[0010] Select multiple multimodal language models trained on the same language model and merge the parameters of the language models;

[0011] Based on the input image, multiple visual encoders are used to extract rich visual features;

[0012] Input text, extract features from the input text through the language model to obtain text features;

[0013] Combine visual features and text features to obtain fusion features;

[0014] The fused features are input into the language model for reasoning, and the reasoning results are output.

[0015] Furthermore, the collecting of image data and raising text questions based on the image include:

[0016] Use high-resolution photography equipment to collect image data;

[0017] After collecting images, textual questions are asked for each image, which will be used for reasoning on multimodal tasks.

[0018] Furthermore, the selecting of multiple multimodal language large models trained based on the same language model includes:

[0019] When selecting a large multimodal model, it is necessary to consider the composition and functionality of the following three key modules: the pre-processing module, the modality-specific visual encoder, and the modality-specific mapping module;

[0020] Among them, the function of the pre-processing module is to standardize and preliminarily convert the input image data so that the subsequent visual encoder can effectively process the image information; the visual encoder is responsible for converting the pre-processed image data into visual features; the mapping module is responsible for converting the visual features extracted by the visual encoder into an embedded format that can be processed by the language model.

[0021] Furthermore, the step of selecting a plurality of multimodal language large models trained based on the same language model also includes:

[0022] After pre-processing, encoding and mapping, the visual features are combined with the text features and input into a shared language model for inference. The parameters of the language model are shared among all multimodal models to ensure the consistency of the language part.

[0023] The language model generates the final reasoning result by processing text and visual information;

[0024] When selecting large multimodal models, it is necessary to ensure that the language model parts of these large multimodal models are based on the same pre-trained model (the parameters are denoted as Θ), that is, the language model parameters of each model remain consistent.

[0025] Furthermore, the parameters of the merged language model include:

[0026] After selecting multiple multimodal large models, the language model parameters from different multimodal large models are merged to generate a unified language model that can interpret the visual information from multiple visual encoders; the specific formula is expressed as:

[0027]

[0028] In the formula, Θ pre represents the parameters of the pre-trained language model, Θ i represents the language model parameters of the i-th multimodal large model, M is the number of multimodal large models, and λ is an adjustable merging coefficient; (Θ i -Θ pre ) is each model relative to its pre-trained language model.

[0029] Furthermore, the method of extracting visual features based on the input image using multiple visual encoders includes:

[0030] Based on the input image x i , use multiple different visual encoders for feature extraction, each visual encoder focuses on extracting different levels of image features;

[0031] The extracted visual features V i Used in combination with the language model as one of the inputs in the multimodal reasoning process.

[0032] Furthermore, the splicing of visual features and text features includes:

[0033] Concatenate multiple visual features:

[0034] V F =[V1,V2,…,V i ]

[0035] Where V F represents the concatenated feature vector, [] represents the concatenation of multiple visual features V i Stitching along a specific dimension.

[0036] Furthermore, the reasoning process of the language model is expressed as follows:

[0037] y=f(V F ,T;Θ)

[0038] Where V F is the concatenated feature vector, T is the text feature, and Θ is the parameter of the language model.

[0039] The second technical solution adopted by the present invention is:

[0040] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement a multimodal model visual perception capability enhancement method as described above.

[0041] The fourth technical solution adopted by the present invention is:

[0042] A computer-readable storage medium, wherein at least one instruction, at least one program, code set or instruction set is stored in the storage medium, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement a multimodal model visual perception capability enhancement method as described above.

[0043] The fifth technical solution adopted by the present invention is:

[0044] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned method for enhancing the visual perception ability of a multimodal model.

[0045] The beneficial effects of the present invention are: the present invention can effectively improve the performance of multimodal tasks without additional training and reduce deployment overhead. In addition, by using the technology of the present invention, the visual perception effect of a large multimodal model can be significantly improved by integrating the advantages of different visual encoders, while reducing the demand for computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1is a flowchart of the steps of a method for enhancing the visual perception ability of a multimodal model in an embodiment of the present invention;

[0048] Figure 2 It is a schematic diagram of a method for enhancing the visual perception ability of a multimodal model in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0050] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0051] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0052] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0053] With the widespread application of large multimodal models in the field of artificial intelligence, improving the visual perception ability of the model has become an important research direction. Existing multimodal models usually rely on the combination of a single visual encoder and a language model. Although this method can complete multimodal tasks to a certain extent, in complex scenes or diverse visual inputs, a single encoder is often difficult to fully capture and parse the rich information in the image. In addition, the existing technology requires the realignment of the visual encoder and the language model and a large amount of training. The training process is costly and it is difficult to effectively respond to multiple visual perception needs. In order to overcome these limitations, the present invention proposes a training-free multimodal model visual perception ability enhancement scheme, which can significantly improve the model's visual perception ability in multimodal tasks without increasing additional training costs. By combining multiple visual encoders fine-tuned based on the same language model and merging the parameters of the language model, the scheme of the present invention achieves excellent performance of large multimodal models in complex tasks.

[0054] Example 1

[0055] like Figure 1 and Figure 2 As shown, this embodiment provides a method for enhancing visual perception ability of a multimodal model without training, which specifically includes the following steps:

[0056] S1. Collect image data and ask text questions based on the image.

[0057] As an optional implementation, step S1 specifically includes the following steps:

[0058] S1-1: First, use a high-resolution photographic device to collect image data. The photographic device can be a digital camera, a smartphone, or a professional photographic system. The user should ensure that the device is in proper working condition and avoid image blur or distortion during the shooting process. All collected images will be used for subsequent multimodal model inference tasks.

[0059] S1-2: When collecting images, users can shoot in different scenes according to their needs, such as indoors, outdoors, under natural light or artificial lighting, etc. The collected images can include any content of interest, such as still life, people, scenery, etc. The collection process does not need to consider the diversity or complexity of the image, and the focus is on obtaining high-quality image data for subsequent testing and analysis of multimodal tasks.

[0060] S1-3: After collecting images, users need to ask text questions for each image, which will be used for reasoning in multimodal tasks. Questions can be freely set based on the image content to ensure that the questions are relevant to the visual content of the image. For example:

[0061] a) If the image contains cars, the user can ask the question: “How many cars are there in the image?”

[0062] b) If the image contains vehicles, the user can ask the question: “How many cars are there in the image?”

[0063] These questions will be used as input to the model together with the image data for subsequent multimodal reasoning.

[0064] S2. Select multiple multimodal language models trained based on the same language model and merge the parameters of the language models.

[0065] In one embodiment, step S2 specifically includes the following steps:

[0066] S2-1: When choosing a multimodal large model (MLLM), it is necessary to consider the composition and functions of the following three key modules: pre-processing module, modality-specific visual encoder, and modality-specific mapping module. These modules work together to effectively fuse the input visual data with the language model to achieve cross-modal reasoning.

[0067] a) Pre-processing module

[0068] The main function of the pre-processing module is to standardize and preliminarily transform the input image data so that the subsequent encoder can effectively process the image information. Specific operations may include image segmentation (such as dividing the image into local areas) or performing some basic pre-processing operations (such as normalization, scaling, etc.). The pre-processing module ensures that the input image is normalized so that the encoder can efficiently extract meaningful visual features from it.

[0069] Mathematically expressed as: x′ i =Preprocessing i (x i ), where x i is the input original image data, x′ i It is the data after preprocessing. i Represents the pre-processing module of the i-th model.

[0070] b) Visual Encoder

[0071] The visual encoder is responsible for converting the pre-processed image data into visual features. Different multimodal models may use different encoder architectures, common ones include convolutional neural networks (CNN) or Transformer-based architectures (such as Vision Transformer, ViT). These visual encoders extract high-dimensional features from images, which represent important information in the image and are used for subsequent cross-modal reasoning.

[0072] The output of the visual encoder is the visual feature V, expressed as: V = Env i (x′ i ) Among them, Env i is the visual encoder of the ith model, x′ i is the input image after pre-processing, and V is the visual feature extracted by the encoder.

[0073] c) Mapper

[0074] The mapping module is responsible for converting the visual features V extracted by the encoder into an embedding format that the language model can process. Since the visual features and the input of the language model are usually in different spaces, the mapping module needs to transform these features so that they can be combined with the text input of the language model for cross-modal reasoning.

[0075] The function of the mapping module is expressed as: V i =Proj i (V) Among them, Proj i is the mapping module of the ith model, V i is the feature extracted from the visual encoder, V i It is the embedded vector after mapping, which is used to combine with the language model.

[0076] d) Language Model

[0077] After pre-processing, encoding and mapping, the visual features are combined with the text features and input into the shared language model for reasoning. The parameters of the language model are shared among all multimodal models to ensure the consistency of the language part. The language model generates the final reasoning result by processing text and visual information. When selecting a large multimodal model, it is necessary to ensure that the language model parts of these models are based on the same pre-trained model (parameters are denoted as Θ), that is, the language model parameters of each model are consistent. The visual encoder and mapping module can be designed differently according to the task requirements.

[0078] e) Diversified model selection

[0079] When choosing a model, give priority to multimodal models with different visual encoders and mapping modules. Different encoders may have different focuses when processing visual data. Some are better at extracting local details, while others are better at extracting global information. By integrating multiple models with different characteristics, the ability to handle complex visual tasks can be improved, and the performance of the model in multimodal tasks can be enhanced.

[0080] S2-2: A single language model cannot be directly aligned with multiple encoders from different multimodal large models (MLLMs) because these models have not been trained for alignment. Direct alignment training of multiple visual encoders usually requires extremely high computational overhead. Therefore, based on existing research methods, simply merging language model parameters from different visual encoders can help the language model interpret the visual information in multiple visual encoders. This merging process does not require retraining and can effectively reduce computational costs.

[0081] After selecting multiple large multimodal models, we merge the language model parameters from different models to generate a unified language model that can interpret the visual information from multiple visual encoders. The specific merging process can be expressed as:

[0082]

[0083] Among them, Θ pre represents the parameters of the pre-trained language model, Θ i represents the language model parameters of the i-th multimodal large model, M is the number of multimodal large models, and λ is an adjustable merging coefficient. (Θ i -Θ pre ) is each model relative to its pre-trained language model.

[0084] S3. Based on the input image, multiple visual encoders are used to extract rich visual features.

[0085] Exemplarily, in this step, the system extracts visual features of the input image through a visual encoder. Each visual encoder receives input image data x i , and convert it into the corresponding visual feature V i The specific processing process is as follows:

[0086] First, input image x i It is passed to the visual encoder, which is responsible for analyzing and extracting features from the image. The functions of the visual encoder include identifying local features, global structures, and other key information in the image. The extracted visual features are represented as:

[0087] V i =Proj i (Enc i (Preprocessing i (x i )))

[0088] Among them, V i Represents the image data x iExtracted visual landmarks. These visual features may contain information about the color, texture, shape, and spatial relationships of the image.

[0089] In order to improve the model's ability to understand complex images, multiple different visual encoders can be used, each focusing on extracting different levels of image features. For example, some encoders may be better at capturing local detail information in an image, while other encoders may focus more on the global features of the image. By working in this way, multiple encoders can generate richer visual feature representations.

[0090] Extracted visual features V i It will then be used in conjunction with the language model in subsequent steps as one of the inputs in the multimodal reasoning process.

[0091] S4. Input text, extract features from the input text through a language model, and obtain text features.

[0092] In some embodiments, the input text t is processed by the word embedding module of the language model to extract text features. Specifically, the input text is converted into a high-dimensional vector representation by the embedding function T=Embedding(t). The vector contains the semantic information of the text.

[0093] S5. Concatenate visual features and text features to obtain fused features.

[0094] The extracted visual features V i It is concatenated with the text feature T to form a multimodal input feature for the subsequent reasoning process. The purpose of the concatenation operation is to combine visual information with text information so that the language model can process the information of these two modalities at the same time and perform multimodal reasoning.

[0095] First, the system receives the visual features V extracted from the visual encoder i The concatenation operation can be expressed as:

[0096] V F =[V1,V2,…,Vi]

[0097] Among them, V F represents the concatenated multimodal feature vector, [] represents the visual feature V i The concatenation is performed along a specific dimension. In this way, visual information and textual information are expressed together in the same input space, ensuring that the language model can comprehensively process input information from both modalities. The concatenated joint feature vector retains the information of both visual and textual modalities and combines them into a complete input representation through the concatenation operation. This concatenation operation not only retains the independence of each modality feature, but also provides richer cross-modal information input for subsequent reasoning.

[0098] S6. Input the fused features into the language model for reasoning and output the reasoning results.

[0099] The concatenated multimodal feature vectors are input into the language model for inference calculation. The language model generates the final cross-modal reasoning result by processing the joint input containing visual features and text features. The purpose of this step is to use the semantic understanding ability of the language model and combine it with visual features for comprehensive reasoning.

[0100] The language model uses its pre-trained parameters Θ merged Process the input and perform a joint interpretation of semantic and visual features. The language model processes the input multimodal information in sequence through its internal multi-layer structure and generates inference results.

[0101] The reasoning process of the language model can be expressed as:

[0102] y=f(V F ,T;Θ)

[0103] Among them, y is the output of the language model.

[0104] In general, existing large multimodal models usually rely only on the alignment of a single visual encoder and a language model to complete multimodal tasks, resulting in the model's visual perception ability being limited to the characteristics of the encoder. This method is prone to insufficient or biased information capture when dealing with complex and diverse visual scenes, and cannot fully utilize the multi-dimensional understanding of image features by different visual encoders. In contrast, the present invention innovatively proposes a multi-visual module integration solution, which enables the model to simultaneously obtain multi-level and multi-perspective visual information by splicing and fusing features from different visual encoders. In addition, the present invention eliminates the need for additional retraining of the visual encoder and language model by merging the language model parameters of multiple large multimodal models. This parameter merging method greatly reduces the computational overhead of realignment and training, and ensures the inference efficiency of the model, so that the model significantly reduces the computational and time costs while maintaining high accuracy.

[0105] By combining multiple visual modules, the present invention can significantly improve visual perception capabilities in multimodal tasks, especially in tasks that require detailed image analysis or complex scene perception. The model can capture local and global features of the image from multiple angles, making the reasoning results more accurate and comprehensive. Secondly, the prior art usually requires complex and time-consuming retraining of the model, while the present invention can achieve enhancement without additional training through the method of parameter merging, which greatly reduces the difficulty of model development and deployment, and also reduces the required computing resources and costs. Third, the deployment cost of the present invention is lower, and the merged model can efficiently process features from multiple visual encoders, and can also run smoothly in resource-constrained environments (such as embedded devices or edge computing scenarios). Finally, the present invention has a wide range of application adaptability, not only suitable for enhancing the visual perception capabilities of multimodal large models, but also can play a role in other multimodal tasks, such as text generation, visual question answering, image recognition, etc., significantly improving the accuracy and performance of these tasks. Therefore, the scheme of the present invention has high technical advantages and wide practical application value.

[0106] Example 2

[0107] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 1 A method for enhancing visual perception capabilities of a multimodal model is shown.

[0108] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0109] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0110] Since the electronic device is an electronic device corresponding to a multimodal model visual perception capability enhancement method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0111] Example 3

[0112] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 A method for enhancing visual perception capabilities of a multimodal model is shown.

[0113] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0114] Since the storage medium is a storage medium corresponding to a method for enhancing visual perception ability of a multimodal model in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0115] Example 4

[0116] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a multimodal model visual perception capability enhancement method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0117] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0119] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for enhancing the visual perception ability of a multimodal model, characterized in that: The following steps are involved: Collect image data and ask text questions based on the image; Select multiple multimodal language models trained on the same language model and merge the parameters of the language models; Based on the input image, multiple visual encoders are used to extract visual features; Input text, extract features from the input text through the language model to obtain text features; Combine visual features and text features to obtain fusion features; The fused features are input into the language model for reasoning, and the reasoning results are output.

2. The method for enhancing the visual perception ability of a multimodal model according to claim 1, characterized in that: The collecting of image data and raising text questions based on the image include: Use high-resolution photography equipment to collect image data; After collecting images, textual questions are asked for each image, which will be used for reasoning on multimodal tasks.

3. The method for enhancing the visual perception ability of a multimodal model according to claim 1, characterized in that: The selecting of multiple multimodal language models trained based on the same language model includes: When choosing a large multimodal model, it is necessary to consider the composition and functions of the following three key modules: pre-processing module, visual encoder, and mapping module; The function of the pre-processing module is to standardize and preliminarily convert the input image data so that the subsequent visual encoder can effectively process the image information; the visual encoder is responsible for converting the pre-processed image data into visual features; The mapping module is responsible for converting the visual features extracted by the visual encoder into an embedding format that the language model can process.

4. The method for enhancing the visual perception ability of a multimodal model according to claim 3, characterized in that: The step of selecting a plurality of multimodal language large models trained based on the same language model also includes: After pre-processing, encoding and mapping, the visual features are combined with the text features and input into a shared language model for inference. The parameters of the language model are shared among all multimodal models to ensure the consistency of the language part. The language model generates the final reasoning result by processing text and visual information; When selecting large multimodal models, it is necessary to ensure that the language model parts of these large multimodal models are based on the same pre-trained model, that is, the language model parameters of each model remain consistent.

5. The method for enhancing the visual perception ability of a multimodal model according to claim 1, characterized in that: The parameters of the merged language model include: After selecting multiple multimodal large models, the language model parameters from different multimodal large models are merged to generate a unified language model that can interpret the visual information from multiple visual encoders; the specific formula is expressed as: In the formula, Θ pre represents the parameters of the pre-trained language model, Θ i represents the language model parameters of the i-th multimodal large model, M is the number of multimodal large models, and λ is an adjustable merging coefficient; (Θ i -Θ pre ) is each model relative to its pre-trained language model.

6. The method for enhancing the visual perception ability of a multimodal model according to claim 1, characterized in that: Based on the input image, multiple visual encoders are used to extract visual features, including: Based on the input image x i , use multiple different visual encoders for feature extraction, each visual encoder focuses on extracting different levels of image features; The extracted visual features V i Used in combination with the language model as one of the inputs in the multimodal reasoning process.

7. The method for enhancing the visual perception ability of a multimodal model according to claim 1, characterized in that: The splicing of visual features and text features includes: Concatenate multiple visual features: V F =[V1,V2,…,V i ] Where V F represents the concatenated feature vector, [] represents the concatenation of multiple visual features V i Stitching along a specific dimension.

8. The method for enhancing the visual perception ability of a multimodal model according to claim 1, characterized in that: The reasoning process of the language model is expressed as follows: y=f(V F ,T;Θ) Where V F is the concatenated feature vector, T is the text feature, and Θ is the parameter of the language model.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

Images