Visual representation compression method and device based on large multi-modal model

Abstract

The invention provides a visual representation compression method based on a large-scale multi-modal model. The method comprises the following steps: compressing a plurality of input first visual tokens into at least one second visual token; before the large-scale multi-modal model is input, the second visual token and the text token are fused, and corresponding fusion information is generated; and inputting the fusion information into the large-scale multi-modal model for processing. The invention further provides a visual representation compression device based on the large multi-modal model, a storage medium and electronic equipment. Therefore, the calculation efficiency can be remarkably improved, the memory occupation is reduced, and the model flexibility is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of large-scale multimodal models, and in particular to a visual representation compression method, device, storage medium and electronic device based on a large-scale multimodal model. Background Art

[0002] In recent years, with the development of large multimodal models (LMMs), models such as GPT-4o have been widely used in real-time interaction scenarios. These models use visual encoders to convert images into visual tokens and process them through large language models (LLMs) to enhance the understanding of visual information. However, existing large multimodal models face many challenges in terms of computational efficiency and processing performance.

[0003] First, existing large multimodal models need to embed a large number of visual tokens into the context of large language models when processing images (Liu et al., 2023). Taking the widely used CLIP ViT-L / 336px as an example, an image is encoded into 576 visual tokens, which greatly increases the computational complexity. Compared with large language models that only process text inputs, large multimodal models need to process not only text tokens but also visual tokens, resulting in a significant increase in overall computational overhead. The problem is exacerbated in the context of high-resolution image and video processing, as these tasks require processing more visual tokens.

[0004] In order to improve the efficiency of large multimodal models, researchers mainly focus on reducing the size of the model or reducing the number of visual tokens. Some methods reduce the total number of parameters by replacing a smaller large language model backbone (Chu et al., 2023), or improve efficiency through quantization techniques. However, these methods often affect the capacity of the large language model backbone, resulting in reduced performance on visual tasks. In addition, existing visual token compression methods mostly rely on predefined rules (Shang et al., 2024), which may lead to the loss of visual information, thus affecting the overall performance of the model. In terms of video processing, existing large multimodal models often reduce the number of visual tokens by reducing the number of video frames (Lin et al., 2023). However, this frame selection or merging method may lose some key frames or misunderstand the timing information of the video, further affecting the effect of video understanding. Therefore, how to improve the computational efficiency of large multimodal models while ensuring performance is an urgent problem that current technology needs to solve.

[0005] Although the existing technology has made some progress in the research of large-scale multimodal models, there are several problems in the design and implementation of large-scale multimodal models, which are mainly reflected in the redundancy of the number of visual tokens, uneven distribution of importance, lack of a unified processing architecture and limitations of post-fusion.

[0006] First, the number of visual tokens in the prior art is often quite redundant, and many large multimodal models use a large number of visual tokens when processing images (Liu et al., 2023). For example, widely used visual encoders such as CLIP ViT-L / 336px encode an image into 576 visual tokens. This redundant number of tokens not only increases the computational complexity, but also leads to a significant increase in memory consumption. Although some methods attempt to perform token compression through visual encoders, these methods usually directly reduce the number of visual tokens and often fail to effectively retain important visual information. As a result, these methods may cause the model's performance on visual tasks to degrade and fail to accurately understand and process visual input.

[0007] Secondly, in the existing large multimodal model architecture, the importance of visual tokens is unevenly distributed. Studies have shown that visual tokens play a vital role in the early layers of the model, as these layers are responsible for fusing visual information with text tokens. However, as the number of layers increases, the subsequent layers pay less attention to visual tokens, which means that in deep models, many visual tokens may not be fully utilized. This uneven importance distribution leads to a waste of computing resources and cannot effectively improve the overall performance of the model. Most existing large multimodal models adopt a late fusion approach, that is, the visual tokens and text tokens are fused after they are input into the large language model, but this approach has certain limitations. First, late fusion may lead to insufficient integration of visual information and text information, affecting the model's understanding ability. Second, late fusion requires processing a large number of visual tokens, further increasing the computational burden. In high-resolution image and long video processing scenarios, the efficiency problem of late fusion is particularly prominent, resulting in delayed response time and affecting user experience.

[0008] In addition, existing research on large-scale multimodal models usually focuses on local optimization and lacks systematic consideration of the overall architecture. Many methods attempt to improve model efficiency by improving a single module, such as through model compression, quantization technology, etc., but this often leads to insufficient adaptability to other parts of the model. As a result, this one-sided optimization method cannot effectively cope with diverse input scenarios. For example, processing different types of inputs (such as high-resolution images and videos) often requires different architectures, and existing technologies lack flexibility in this regard, limiting the scope of application of the model.

[0009] In summary, the existing technology obviously has inconveniences and defects in practical use, so it is necessary to improve it. Summary of the invention

[0010] In view of the above-mentioned defects, the object of the present invention is to provide a visual representation compression method, device, storage medium and electronic device based on a large multimodal model, which can significantly improve computing efficiency, reduce memory usage and enhance model flexibility.

[0011] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0012] In a first aspect, an embodiment of the present invention provides a visual representation compression method based on a large multimodal model, comprising:

[0013] compressing the inputted plurality of first visual tokens into at least one second visual token;

[0014] Before inputting into the large multimodal model, fusing the second visual token with the textual token to generate corresponding fusion information;

[0015] The fused information is input into the large multimodal model for processing.

[0016] According to the visual representation compression method based on a large multimodal model of the present invention, the step of compressing the input multiple first visual tokens into at least one second visual token comprises:

[0017] Through an adaptive feature selection algorithm, multiple input first visual tokens are compressed into at least one second visual token, and the number of the second visual tokens is less than the number of compressed first visual tokens.

[0018] According to the large multimodal model-based visual representation compression method of the present invention, the goal of the adaptive feature selection algorithm includes reducing the number of visual tokens occupied by the visual representation while retaining key information of the visual tokens.

[0019] According to the visual representation compression method based on a large multimodal model of the present invention, the step of compressing the input multiple first visual tokens into at least one second visual token comprises:

[0020] When processing a high-resolution image, dynamically adjusting the number of the second visual tokens according to the resolution and complexity of the high-resolution image to ensure that the large multimodal model can maintain a balance between performance and memory consumption when processing the high-resolution image; and / or

[0021] Through training and optimization, the number of the second visual tokens is further reduced while maintaining a predetermined performance.

[0022] According to the visual representation compression method based on a large multimodal model of the present invention, the number of the first visual tokens is 576, and the number of the second visual tokens is 1.

[0023] According to the visual representation compression method based on a large multimodal model of the present invention, before inputting the large multimodal model, the step of fusing the second visual token with the text token to generate corresponding fusion information includes:

[0024] Before inputting into the large multimodal model, the second visual token is fused with the text token to generate the corresponding fused information, and the contextual relationship of the fused information is enhanced in combination with the attention mechanism.

[0025] According to the visual representation compression method based on a large multimodal model of the present invention, the step of inputting the fusion information into the large multimodal model for processing comprises:

[0026] The fused information is input into the large multimodal model, and the large multimodal model processes different types of fused information through a unified architecture, wherein the fused information includes ordinary images, high-resolution images, and videos.

[0027] In a second aspect, an embodiment of the present invention provides a visual representation compression device based on a large multimodal model constructed based on any one of the methods described above, comprising:

[0028] a visual representation compression module, configured to compress a plurality of input first visual tokens into at least one second visual token;

[0029] A multimodal pre-fusion module, used for fusing the second visual token with the text token to generate corresponding fusion information before inputting into the large multimodal model;

[0030] The multimodal processing module is used to input the fusion information into the large multimodal model for processing.

[0031] In a third aspect, an embodiment of the present invention provides a computer program for storing a method for compressing visual representations based on a large multimodal model as described in any one of the above.

[0032] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements any one of the visual representation compression methods based on a large multimodal model when executing the computer program.

[0033] The visual representation compression technology based on the large multimodal model of the present invention includes: first, compressing multiple first visual tokens input into at least one second visual token, preferably, compressing 576 first visual tokens into 1 second visual token, while retaining visual information to the maximum extent. By significantly reducing the number of visual tokens, the method effectively reduces the computational burden and memory usage, and improves the operating efficiency of the model. Second, before inputting the large multimodal model, the second visual token is fused with the text token to generate corresponding fusion information; effectively reducing 77% of the computational workload, achieving a response time as low as 40 milliseconds of delay, and significantly reducing the video memory usage of each image from 360MB to 0.6MB, thereby improving the real-time interactive capability of the model. Third, the fusion information is input into the large multimodal model for processing. Preferably, the large multimodal model processes different types of fusion information through a unified architecture, and the fusion information includes ordinary images, high-resolution images and videos. The design of the unified architecture enhances the applicability and flexibility of the model, reduces the switching requirements between different models, and improves the overall processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of a visual representation compression method based on a large multimodal model provided in Embodiment 1 of the present invention;

[0035] Figure 2 It is a flowchart of a visual representation compression method based on a large multimodal model provided in Embodiment 2 of the present invention;

[0036] Figure 3 is a structural schematic diagram of a visual representation compression device based on a large multimodal model provided in Embodiment 1 of the present invention;

[0037] Figure 4 is a schematic structural diagram of a visual representation compression device based on a large multimodal model provided in Embodiment 2 of the present invention;

[0038] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] It should be noted that references to "one embodiment", "embodiment", "example embodiment", etc. in this specification refer to the embodiment described, which may include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not refer to the same embodiment. Furthermore, when describing specific features, structures or characteristics in conjunction with an embodiment, whether or not there is an explicit description, it has been shown that it is within the knowledge of those skilled in the art to combine such features, structures or characteristics into other embodiments.

[0041] In addition, certain words are used in the specification and subsequent claims to refer to specific components or parts. Those with ordinary knowledge in the relevant field should understand that manufacturers can use different nouns or terms to refer to the same component or part. This specification and subsequent claims do not use differences in names as a way to distinguish components or parts, but use differences in the functions of components or parts as the criteria for distinction. "Including" and "including" mentioned throughout the specification and subsequent claims are open-ended terms and should be interpreted as "including but not limited to". In addition, the word "connected" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connection through other devices.

[0042] The visual representation compression method based on a large multimodal model provided by an embodiment of the present invention is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0043] In view of the shortcomings of existing large-scale multimodal models in processing visual tokens, the present invention proposes a visual representation compression method based on a large-scale multimodal model, aiming to effectively reduce the number of tokens occupied by the visual representation. Through a hierarchical analysis of the large-scale multimodal model architecture, the inventors found that the role of visual tokens in the early layers of the model is particularly important, while the attention in the later layers is significantly reduced. This discovery provides an optimization idea for the present invention, that is, by fusing the visual information before the large-scale multimodal model, the number of visual tokens input to the large-scale multimodal model is reduced without losing model performance.

[0044] The present invention introduces a multimodal fusion module that fuses visual information and textual information before the large multimodal model, ensuring that the visual information has been fully processed before being input into the large language model. This allows the model to maintain comparable performance using only one visual token, demonstrating that visual information can still be efficiently transmitted under extreme compression.

[0045] In addition, the present invention proposes a unified architecture that can flexibly handle different types of inputs, including ordinary images, high-resolution images, and videos. The design of this architecture takes into account the characteristics of different tasks and can achieve efficient multimodal understanding in a variety of scenarios.

[0046] In summary, the technical difficulty of the present invention lies in how to compress visual tokens to a greater extent while ensuring the integrity of visual information and the stability of model performance. By introducing pre-fusion modules and unified architecture, the model not only improves computational efficiency, but also provides stronger support for real-time multimodal interaction, showing its broad prospects in future applications.

[0047] Figure 1 : is a flow chart of a visual representation compression method based on a large multimodal model provided in Embodiment 1 of the present invention, the method comprising:

[0048] Step S101: compressing multiple input first visual tokens into at least one second visual token.

[0049] Preferably, the number of the first visual tokens is 576, and the number of the second visual token is 1. That is, the input 576 first visual tokens are compressed into 1 second visual token through the adaptive feature selection algorithm.

[0050] The present invention designs a visual representation compression method, which processes the input visual information (such as images or video frames) and compresses the image traditionally encoded as 576 visual tokens into only 1 visual token through an adaptive feature selection algorithm. In this process, the system can retain key information to the maximum extent and avoid the information loss common in traditional methods, thereby ensuring the stability of model performance.

[0051] Step S102, before inputting into the large multimodal model, the second visual token is fused with the text token to generate corresponding fusion information.

[0052] The present invention introduces a text-visual multimodal pre-fusion method, preferably by combining the attention mechanism, the system effectively fuses the compressed visual tokens with the text tokens at the input stage. This pre-fusion can not only enhance the contextual relationship of the information, but also significantly improve the processing efficiency. Experimental results show that the inference time can be shortened to less than 40 milliseconds, which can significantly improve the user experience.

[0053] Step S103: input the fused information into a large multimodal model for processing.

[0054] Preferably, the fusion information is input into a large multimodal model, and the large multimodal model processes different types of fusion information through a unified architecture, and the fusion information includes ordinary images, high-resolution images and videos. The unified architecture of the present invention can process images, high-resolution images and videos at the same time. The architecture flexibly responds to the processing requirements of different modalities and improves the versatility of the model by sharing parameters and modules. When faced with high-resolution images, the system can dynamically adjust the number of visual tokens to ensure that memory consumption is reduced while ensuring performance, effectively reducing the occupancy of GPU video memory.

[0055] The present invention aims to achieve efficient visual representation processing and multimodal information fusion through a series of innovative technical means, and finally obtain an efficient and unified large-scale multimodal model. It solves a series of technical problems of existing large-scale multimodal models in visual information processing, which are mainly reflected in the redundancy of the number of visual tokens, uneven distribution of importance, lack of a unified processing architecture and limitations of late fusion. These problems lead to a significant increase in computational overhead and excessive memory consumption, which affects the response speed and performance of the model when processing high-resolution images and videos. Although the existing technology has improved the efficiency of the model to a certain extent, it has failed to fundamentally solve the problems of the number of tokens and insufficient information fusion, limiting the real-time and effectiveness of multimodal interaction.

[0056] The key technical points and beneficial effects of the present invention are as follows:

[0057] (1) Visual representation compression method. The 576 visual tokens in the existing technology are compressed to only 1 visual token, while retaining the visual information to the maximum extent. By significantly reducing the number of visual tokens, this method effectively reduces the computational burden and memory usage, and improves the operating efficiency of the model.

[0058] (2) Text-vision multimodal pre-fusion method. This method effectively reduces the amount of computation by 77%, achieves a response time as low as 40 milliseconds, and significantly reduces the video memory usage of each image from 360MB to 0.6MB, thereby improving the model's real-time interactive capabilities.

[0059] (3) Unified image, high-resolution image, and video modeling architecture. The unified architecture can process images, high-resolution images, and videos simultaneously, and can handle multiple input types with a single model. The design of this architecture enhances the applicability and flexibility of the model, reduces the need to switch between different models, and improves overall processing efficiency.

[0060] Figure 2 : is a flow chart of a visual representation compression method based on a large multimodal model provided in Embodiment 2 of the present invention, the method comprising:

[0061] Step S201 : compressing multiple input first visual tokens into at least one second visual token through an adaptive feature selection algorithm, and the number of the second visual tokens is less than the number of compressed first visual tokens.

[0062] Preferably, the goal of the adaptive feature selection algorithm includes reducing the number of visual tokens occupied by the visual representation while retaining key information of the visual tokens.

[0063] Preferably, when processing high-resolution images, the number of second visual tokens is dynamically adjusted according to the resolution and complexity of the high-resolution images to ensure that the large multimodal model can maintain a balance between performance and memory consumption when processing high-resolution images.

[0064] Preferably, through training and optimization, the number of second visual tokens is further reduced while maintaining a predetermined performance.

[0065] Preferably, the number of the first visual tokens is 576, and the number of the second visual token is 1. That is, the input 576 first visual tokens are compressed into 1 second visual token through the adaptive feature selection algorithm.

[0066] Step S202, before inputting into the large multimodal model, the second visual token is fused with the text token to generate corresponding fusion information, and the contextual relationship of the fusion information is enhanced in combination with the attention mechanism.

[0067] Step S203, inputting the fused information into a large multimodal model, the large multimodal model processes different types of fused information through a unified architecture, the fused information including ordinary images, high-resolution images and videos.

[0068] The visual representation compression method based on a large multimodal model of the present invention achieves the goals of significantly improving computational efficiency, reducing memory usage, enhancing model flexibility, and improving user experience through visual representation compression, text-visual multimodal pre-fusion, and a unified modeling architecture. Specifically, by reducing the number of visual tokens, the computational complexity and response time of the model during the reasoning process are reduced, while the demand for video memory is significantly reduced, so that complex visual tasks can still be efficiently processed in a resource-limited environment. In addition, the unified processing architecture improves the model's adaptability to a variety of input types, providing a good foundation for the widespread promotion of multimodal applications, which has obvious advantages in practical applications.

[0069] It should be noted that the visual representation compression method based on a large multimodal model provided in the embodiment of the present invention can be executed by an electronic device, a device, or a control module in the device for executing the visual representation compression method based on a large multimodal model. In the embodiment of the present invention, the visual representation compression method based on a large multimodal model is executed by a visual representation compression device based on a large multimodal model as an example to illustrate the visual representation compression device based on a large multimodal model provided in the embodiment of the present invention.

[0070] Figure 3 1 is a schematic diagram of the structure of a visual representation compression device based on a large multimodal model provided in Embodiment 1 of the present invention. The visual representation compression device 100 based on a large multimodal model includes a visual representation compression module 10, a multimodal pre-fusion module 20 and a multimodal processing module 30, wherein:

[0071] The visual representation compression module 10 is used to compress the inputted multiple first visual tokens into at least one second visual token. Preferably, the visual representation compression module 10 is used to compress 576 first visual tokens into one second visual token.

[0072] The multimodal pre-fusion module 20 is used to fuse the second visual token with the text token before inputting into the large multimodal model to generate corresponding fusion information.

[0073] The multimodal processing module 30 is used to input the fusion information into a large multimodal model for processing.

[0074] Figure 4 1 is a schematic diagram of the structure of a visual representation compression device based on a large multimodal model provided in Embodiment 2 of the present invention. The visual representation compression device 100 based on a large multimodal model includes a visual representation compression module 10, a multimodal pre-fusion module 20 and a multimodal processing module 30, wherein:

[0075] The visual representation compression module 10 is used to compress the input multiple first visual tokens into at least one second visual token. The visual representation compression module 10 further includes a visual compression submodule 11, a dynamic adjustment submodule 12, and a training optimization submodule 13, wherein:

[0076] The visual compression submodule 11 is used to compress the input multiple first visual tokens into at least one second visual token through an adaptive feature selection algorithm, and the number of the second visual tokens is less than the number of compressed first visual tokens. Preferably, the goal of the adaptive feature selection algorithm includes reducing the number of visual tokens occupied by the visual representation while retaining the key information of the visual tokens. Preferably, the number of the first visual tokens is 576, and the number of the second visual tokens is 1. That is, the visual compression submodule 11 is used to compress 576 first visual tokens into 1 second visual token.

[0077] The dynamic adjustment submodule 12 is used to dynamically adjust the number of second visual tokens according to the resolution and complexity of the high-resolution image when processing the high-resolution image, so as to ensure that the large multimodal model can maintain a balance between performance and memory consumption when processing the high-resolution image.

[0078] The training optimization submodule 13 is used to further reduce the number of second visual tokens while maintaining a predetermined performance through training and optimization.

[0079] The multimodal pre-fusion module 20 is used to fuse the second visual token with the text token before inputting the large multimodal model to generate corresponding fusion information. Preferably, before inputting the large multimodal model, the second visual token is fused with the text token to generate corresponding fusion information, and the contextual relationship of the fusion information is enhanced in combination with the attention mechanism.

[0080] The multimodal processing module 30 is used to input the fusion information into a large multimodal model for processing. Preferably, the fusion information is input into a large multimodal model, and the large multimodal model processes different types of fusion information through a unified architecture, and the fusion information includes ordinary images, high-resolution images and videos.

[0081] The visual representation compression device based on a large multimodal model provided by the embodiment of the present invention can achieve Figures 1-2 The various processes implemented in the embodiment of the visual representation compression method based on a large multimodal model are not described here in detail to avoid repetition.

[0082] The visual representation compression device based on the large multimodal model provided by the embodiment of the present invention includes: first, a visual representation compression module, which is used to compress the input multiple first visual tokens into at least one second visual token, preferably, compress 576 first visual tokens into 1 second visual token, while retaining the visual information to the maximum extent. By significantly reducing the number of visual tokens, the method effectively reduces the computational burden and memory usage, and improves the operating efficiency of the model. Second, a multimodal pre-fusion module, which is used to fuse the second visual token with the text token before inputting the large multimodal model to generate corresponding fusion information; effectively reduce 77% of the computational amount, achieve a response time as low as 40 milliseconds of delay, and significantly reduce the video memory usage of each image from 360MB to 0.6MB, thereby improving the real-time interactive capability of the model. Third, a multimodal processing module, which is used to input the fusion information into the large multimodal model for processing. Preferably, the large multimodal model processes different types of fusion information through a unified architecture, and the fusion information includes ordinary images, high-resolution images and videos. The design of the unified architecture enhances the applicability and flexibility of the model, reduces the switching requirements between different models, and improves the overall processing efficiency.

[0083] The present invention also provides a storage medium for storing Figure 1-2 A computer program for any of the visual representation compression methods based on large multimodal models. For example, a computer program instruction, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer, and can achieve the same technical effect. To avoid repetition, it will not be repeated here. The program instructions for calling the method of the present invention may be stored in a fixed or removable storage medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium and / or stored in a storage medium of a computer device that runs according to the program instructions.

[0084] According to one embodiment of the present invention, the present invention also provides a Figure 5The electronic device 400 shown in the figure may optionally include a storage medium 200 for storing a computer program and a processor 300 for executing the computer program, wherein when the computer program is executed by the processor 300, any of the above-mentioned visual representation compression methods based on a large multimodal model is implemented, triggering the electronic device 400 to execute the methods and / or technical solutions based on the aforementioned multiple embodiments, and the same technical effects can be achieved. To avoid repetition, it is not repeated here. It should be noted that the electronic devices in the embodiments of the present invention include mobile electronic devices and non-mobile electronic devices. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, a super mobile personal computer, a netbook or a personal digital assistant, etc., and the non-mobile electronic device may be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiments of the present invention.

[0085] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0086] The present invention can be implemented on a computer as a computer-implemented method, or implemented in dedicated hardware, or implemented in a combination of the two. The executable code or part thereof for the method according to the present invention can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Optionally, the computer program product includes a non-temporary program code component stored on a computer-readable medium so as to perform the method according to the present invention when the program product is executed on a computer.

[0087] In an alternative embodiment, the computer program comprises computer program code means adapted to perform all the steps of the method according to the invention when the computer program is run on a computer. Optionally, the computer program is embodied on a computer readable medium.

[0088] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0089] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A visual representation compression method based on a large multimodal model, characterized in that: include: compressing the inputted plurality of first visual tokens into at least one second visual token; Before inputting into the large multimodal model, fusing the second visual token with the textual token to generate corresponding fusion information; The fused information is input into the large multimodal model for processing.

2. The method for compressing visual representations based on a large multimodal model according to claim 1, characterized in that: The step of compressing the inputted plurality of first visual tokens into at least one second visual token comprises: Through an adaptive feature selection algorithm, multiple input first visual tokens are compressed into at least one second visual token, and the number of the second visual tokens is less than the number of compressed first visual tokens.

3. The method for compressing visual representations based on a large multimodal model according to claim 2, characterized in that: The goal of the adaptive feature selection algorithm includes reducing the number of visual tokens occupied by the visual representation while retaining the key information of the visual tokens.

4. The method for compressing visual representations based on a large multimodal model according to claim 1, characterized in that: The step of compressing the inputted plurality of first visual tokens into at least one second visual token comprises: When processing a high-resolution image, dynamically adjusting the number of the second visual tokens according to the resolution and complexity of the high-resolution image to ensure that the large multimodal model can maintain a balance between performance and memory consumption when processing the high-resolution image; and / or Through training and optimization, the number of the second visual tokens is further reduced while maintaining a predetermined performance.

5. The method for compressing visual representations based on a large multimodal model according to claim 1, characterized in that: The number of the first visual tokens is 576, and the number of the second visual tokens is 1.

6. The method for compressing visual representations based on a large multimodal model according to claim 1, characterized in that: The step of fusing the second visual token with the text token to generate corresponding fusion information before inputting into the large multimodal model comprises: Before inputting into the large multimodal model, the second visual token is fused with the text token to generate the corresponding fused information, and the contextual relationship of the fused information is enhanced in combination with the attention mechanism.

7. The method for compressing visual representations based on a large multimodal model according to claim 1, characterized in that: The step of inputting the fusion information into the large multimodal model for processing comprises: The fused information is input into the large multimodal model, and the large multimodal model processes different types of fused information through a unified architecture, wherein the fused information includes ordinary images, high-resolution images, and videos.

8. A visual representation compression device based on a large multimodal model constructed based on the method described in any one of claims 1 to 7, characterized in that: include: a visual representation compression module, configured to compress a plurality of input first visual tokens into at least one second visual token; A multimodal pre-fusion module, used for fusing the second visual token with the text token to generate corresponding fusion information before inputting into the large multimodal model; The multimodal processing module is used to input the fusion information into the large multimodal model for processing.

9. A storage medium, characterized in that: Used to store a computer program for executing the visual representation compression method based on a large multimodal model as described in any one of claims 1 to 7.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the visual representation compression method based on a large multimodal model described in any one of claims 1 to 7 is implemented.

Images