Multi-modal information generation method, device and equipment based on visual regression model
Through the multimodal information generation method based on the visual regression model, the pre-trained visual language model is used to process images and text, and the problems of pre-training quality limitations and insufficient inference capabilities of vision-language models in multimodal tasks in the prior art are solved, and more powerful multimodal data processing and inference capabilities are achieved.
Patent Information
- Application Number
- CN202411780537.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-03-21
AI Technical Summary
The existing multimodal large language model has problems with pre-training quality limitations and lack of multimodal reasoning capabilities in visual-language tasks.
Using a multimodal information generation method based on a visual regression model, the images and/or text to be processed are obtained through a pre-trained visual language model, multi-scale image encoding and token encoding are performed, and the VLM backbone network is used for prediction to generate target images and/or target text.
The processing and inference capabilities of multimodal data are realized, and the performance of vision-language models in multimodal tasks is improved.
Smart Images
Figure CN119672468A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of visual language models, and in particular to a method, device and equipment for generating multimodal information based on a visual regression model. Background Art
[0002] In recent years, the emergence of large language models (LLM) such as ChatGPT and LLaMA has caused a huge response in the field of natural language processing. All kinds of tasks in the field of natural language processing can be solved by LLM without the need to design and retrain models for each task. However, in real life, the information people are exposed to is often multi-modal, such as text, vision, speech, etc. Researchers have begun to expand the understanding ability of LLM to multiple modalities and build a multimodal large language model (MLLM).
[0003] However, the current multimodal large language model has inherent defects: the visual encoder and visual generator are pre-trained separately, and their pre-training quality limits the potential of visual-language MLLM; LLM is essentially still processing text information and does not have multimodal reasoning capabilities. Summary of the invention
[0004] Based on this, it is necessary to provide a multimodal information generation method, device and equipment based on a visual regression model to address the above-mentioned technical problems, which can at least form multimodal data processing and reasoning capabilities through a pre-trained visual language model.
[0005] In a first aspect, the present application provides a method for generating multimodal information based on a visual regression model, comprising: obtaining an image and / or text to be processed;
[0006] Inputting the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text associated with the image and / or text to be processed;
[0007] Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0008] In one embodiment, the image to be processed is processed based on a pre-trained target visual language model by the following steps:
[0009] Inputting the image to be processed into a pre-trained multi-scale image encoder to obtain a multi-scale image embedding feature, wherein the multi-scale image embedding feature is used to characterize the image to be processed;
[0010] Determine the image token to be processed by using the maximum distance similarity criterion in the multimodal vocabulary according to the multi-scale image embedding features;
[0011] Two-dimensional position coding is used to positionally encode the image token to be processed;
[0012] Input the to-be-processed image token and / or text token into a pre-trained VLM backbone network, adopt a strategy of predicting the next token of the target text and predicting the next-scale full token of the target image, and obtain a target text token and / or a target image multi-scale token;
[0013] The multi-scale image token is input into a pre-trained multi-scale image decoder to obtain a target image.
[0014] In one embodiment, the step of position encoding the multimodal target image token using two-dimensional position encoding includes:
[0015] In the first dimension position coding, the same coding is used for target image tokens of different scales of an image, and each token of text and special symbols occupies a separate position coding;
[0016] In the second-dimensional position coding, the scale information of the target image token is encoded, and each sub-image token at the same scale has the same position coding, and text and special symbols use the same position coding.
[0017] In one embodiment,
[0018] In one embodiment, the text to be processed is processed based on a pre-trained target visual language model by the following steps:
[0019] Determine the token of the text to be processed in the multimodal vocabulary based on the text to be processed
[0020] Input the to-be-processed text token and the to-be-processed image token into a pre-trained VLM backbone network, and adopt the next text token strategy to obtain a target text token;
[0021] The target text tokens are converted according to the multimodal vocabulary to obtain the target text.
[0022] In one embodiment, the image to be processed is processed based on a pre-trained target visual language model by the following steps:
[0023] The image to be processed is input into a multi-scale image encoder to obtain a multi-scale embedding feature of the image to be processed, and a maximum distance similarity criterion is used to determine a token of the image to be processed using a multimodal vocabulary.
[0024] The image token to be processed and the text token to be processed are input into the pre-trained VLM backbone network, and the target image token is obtained by adopting the strategy of predicting the next-scale full token of the target image.
[0025] The target image token is converted according to the multimodal vocabulary to obtain a multi-scale embedded feature representation of the target image, which is input into a multi-scale image decoder to obtain the target image.
[0026] In one embodiment, the method further comprises:
[0027] The number of tokens for each target image is pre-set;
[0028] After the VLM backbone network outputs the target image start token, a full-token strategy for predicting the next scale of the target image is adopted. After the output of target image tokens at all scales is completed, the VLM backbone network automatically adds an image end token, where the number of scales and the number of tokens for each scale are pre-set based on the multi-scale image encoder and multi-scale image decoder.
[0029] In the second aspect, the present application also provides a multimodal information generation device based on a visual regression model, including an image and text acquisition module and an image and text generation module, the image and text acquisition module is used to acquire the image and / or text to be processed; the image and text generation module is used to input the image and / or text to be processed into a pre-trained target visual language model to obtain the target image and / or target text; wherein the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0031] Obtaining images and / or texts to be processed;
[0032] Inputting the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text associated with the image and / or text to be processed;
[0033] Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:
[0035] Obtaining images and / or texts to be processed;
[0036] Inputting the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text associated with the image and / or text to be processed;
[0037] Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0038] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:
[0039] Obtaining images and / or texts to be processed;
[0040] Inputting the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text associated with the image and / or text to be processed;
[0041] Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0042] The above-mentioned multimodal information generation method, device and equipment based on visual regression model form multimodal data processing and reasoning capabilities through a pre-trained target visual language model. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0044] Figure 1 is a flow chart of a method for generating multimodal information based on a visual regression model in one embodiment;
[0045] Figure 2 is a schematic diagram of a visual language model in one embodiment;
[0046] Figure 3 is a structural block diagram of a multimodal information generating device based on a visual regression model in one embodiment;
[0047] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment.
[0048] Reference numerals and descriptions:
[0049] 10. Image and text acquisition module; 20. Image and text generation module. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0051] The distributed relational database testing method provided in the embodiment of the present application can be applied to various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.
[0052] Please refer to Figure 1 In an exemplary embodiment, a method for generating multimodal information based on a visual regression model is provided, comprising the following steps S101 to S102. In which:
[0053] Step S101, obtaining an image and / or text to be processed.
[0054] Step S102: input the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text associated with the image and / or text to be processed.
[0055] Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0056] The VLM backbone network is the Vision-Language Model (VLM): a technology that combines image and natural language processing. Its main purpose is to understand and interpret the relationship between images and texts, and generate accurate and vivid language descriptions or visual images based on the associated information.
[0057] Among them, the multi-scale image encoder and multi-scale decoder can form a VQ-VAE model.
[0058] Specifically, a multi-scale VQ-VAE model can be trained based on large-scale image data, so that the multi-scale VQ-VAE model can represent the image as a set of multi-scale codewords, and codewords of different scales share a codebook.
[0059] Here, Vector Quantized Variational AutoEncoder (VQ-VAE): is a deep learning model for efficient data compression and generation, especially when large amounts of data need to be processed or implicit representation is required. The predecessor of the VQ-VAE model is the autoencoder (AE) and the variational autoencoder (VAE). The autoencoder (AE) is a representation learning algorithm that can be used for data dimensionality reduction and feature extraction. The variational autoencoder (VAE) normalizes the representation space based on the autoencoder (AE) to ensure that sampling can be performed in the representation space for data generation. VQ-VAE constructs the representation space as a discrete codebook to facilitate autoregressive data generation.
[0060] Multi-scale VQ-VAE encodes images into multi-scale discrete codewords, and codewords of different scales share a codebook; in the image modeling process, VAR receives low-resolution scale codewords as input and autoregressively predicts high-resolution scale codewords. This processing method is better than autoregressive prediction of single-scale VQ-VAE codewords because single-scale codewords do not have a one-dimensional arrangement order like natural language, and there is information leakage between codewords.
[0061] Specifically, the image to be processed can be processed based on a pre-trained target visual language model through the following steps: the image to be processed is input into a multi-scale image encoder to obtain a multi-scale embedding feature of the image to be processed, and the image token to be processed is determined using a multimodal vocabulary using a maximum distance similarity criterion; the image token to be processed and the text token to be processed are input into a pre-trained VLM backbone network, and a strategy of predicting the next-scale full token of the target image is adopted to obtain a target image token; the target image token is converted according to a multi-modal vocabulary to obtain a multi-scale embedding feature representation of the target image, and input into a multi-scale image decoder to obtain a target image.
[0062] Here, the LLM model structure can be used to build the VLM backbone network. The VLM backbone network consists of multiple layers of causal Transformer (Casual Transformer). In the Casual Transformer, the information of all the tokens before the position can be used to predict the next token.
[0063] Specifically, the step of position encoding the multimodal target image tokens using two-dimensional position coding includes: in the first dimension position coding, the same coding is used for target image tokens of different scales of an image, and each token of text and special symbols occupies a separate position coding; in the second dimension position coding, the scale information of the target image token is encoded, each sub-image token at the same scale has the same position coding, and the text and special symbols use the same position coding.
[0064] In this way, using the above two-dimensional position encoding method is more conducive to VLM utilizing information of different modalities and information of different scales of visual modalities.
[0065] Specifically, the image to be processed can also be processed based on a pre-trained target visual language model through the following steps: The image to be processed is input into a multi-scale image encoder to obtain a multi-scale embedding feature of the image to be processed, and the image token to be processed is determined using a multimodal vocabulary using a maximum distance similarity criterion; The image token to be processed and the text token to be processed are input into a pre-trained VLM backbone network, and a strategy of predicting the next-scale full token of the target image is adopted to obtain a target image token; The target image token is converted according to a multi-modal vocabulary to obtain a multi-scale embedding feature representation of the target image, and input into a multi-scale image decoder to obtain a target image.
[0066] Specifically, the text to be processed can be processed based on a pre-trained target visual language model through the following steps: determining the text token to be processed in the multimodal vocabulary according to the text to be processed; inputting the text token to be processed and the image token to be processed into a pre-trained VLM backbone network, and adopting the next text token strategy to obtain the target text token; converting the target text token according to the multimodal vocabulary to obtain the target text.
[0067] The VLM backbone network can be trained by the following steps: pre-training using large-scale visual-language multimodal data; and using instruction fine-tuning technology to enhance the instruction-following capability of the VLM backbone network to obtain a trained target VLM backbone network.
[0068] In this way, for the trained VLM, the instruction fine-tuning technology is used to enhance the VLM's instruction-following ability, especially for the image-text comprehension task, the text-image task and the image editing task, to further improve the VLM's multimodal information processing ability. After the instruction fine-tuning, the VLM is subjected to reinforcement learning based on human feedback to further improve the generation effect and safety of the VLM.
[0069] As an example, see Figure 2In an exemplary embodiment, a visual language model is provided, including a multi-scale VQ-VAE encoder 201, a Tokenizer 202, a position encoding unit 203, a VLM backbone network 204, a multi-scale token 205 and a multi-scale VQ-VAE decoder 206.
[0070] Among them, the multi-scale VQ-VAE model can represent the image as a set of multi-scale codewords, and codewords of different scales share a codebook. The multi-scale VQ-VAE codebook is used to expand the natural language vocabulary, and special tokens " / pic" and "pic / " are added to the vocabulary. " / pic" and "pic / " represent the start and end of the visual token respectively. At this time, the vocabulary can represent the visual-language multimodal information. When using the multi-modal vocabulary for tokenization, the image first passes through the encoder of the multi-scale VQ-VAE to obtain the multi-scale embedding features. According to the multi-scale embedding features, the corresponding token is found in the multi-modal vocabulary using the maximum distance similarity criterion, and the special tokens " / pic" and "pic / " are added to the head and tail of the obtained multi-scale token. VLM consists of multiple layers of causal Transformer (Casual Transformer). In the Casual Transformer, the information of all tokens before the position can be used to predict the next token. When processing text token prediction, the strategy of predicting the next token is still adopted; when processing image token prediction, the strategy of predicting all tokens of the next scale is adopted. At this time, all tokens of the next scale are regarded as a whole, and the information between tokens in the whole can be used mutually, which is in line with the human visual perception process from coarse to fine. Two-dimensional position encoding is used to encode the position of multimodal tokens. In the first dimension position encoding, tokens of different scales of the same image use the same encoding, and each token of text and special symbols occupies a separate position encoding; in the second dimension position encoding, the scale information of image tokens is encoded, and each token at the same scale has the same position encoding, and text and special symbols use the same position encoding. Using this two-dimensional position encoding method, it is more conducive to the VLM backbone network to utilize information of different modalities and information of different scales of visual modalities. For the output generated multi-scale image tokens, the multi-scale image tokens are input to the multi-scale VQ-VAE decoder to explicitly construct the image or text.
[0071] Optionally, the method further includes: presetting the number of tokens for each target image; after the VLM backbone network outputs the target image start token, adopting a strategy of predicting the next scale full token of the target image, and after all scale target image tokens are output, the VLM backbone network automatically adds an image end token, wherein the number of scales and the number of tokens for each scale are pre-set based on the multi-scale image encoder and the multi-scale image decoder.
[0072] As an example, special image identification tokens “ / pic” and “pic / ” can be added to the vocabulary. “ / pic” and “pic / ” represent the start and end of a visual token, respectively. In this case, the vocabulary can represent visual-linguistic multimodal information.
[0073] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0074] Based on the same inventive concept, the embodiment of the present application also provides a multimodal information generation device based on a visual regression model and a multimodal information generation method based on a visual regression model. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the multimodal information generation device based on a visual regression model provided below can be referred to the limitations of the multimodal information generation method based on a visual regression model above, and will not be repeated here.
[0075] Please refer to Figure 3 In an exemplary embodiment, a multimodal information generation device based on a visual regression model is provided, comprising: an image and text acquisition module 10 and an image and text generation module 20, wherein the image and text acquisition module 10 is used to acquire an image and / or text to be processed; the image and text generation module 20 is used to input the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text; wherein the visual language model comprises a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0076] Each module in the above-mentioned multimodal information generation device based on visual regression model can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0077] In an exemplary embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a multimodal information generation method based on a visual regression model is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.
[0078] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0079] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0080] Obtaining images and / or texts to be processed;
[0081] Inputting the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text associated with the image and / or text to be processed;
[0082] Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0083] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0084] Obtaining images and / or texts to be processed;
[0085] Inputting the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text associated with the image and / or text to be processed;
[0086] Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0087] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0088] Obtaining images and / or texts to be processed;
[0089] Inputting the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text associated with the image and / or text to be processed;
[0090] Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
[0091] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0092] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0093] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A multimodal information generation method based on a visual regression model, characterized in that: include: Obtaining images and / or texts to be processed; Inputting the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text associated with the image and / or text to be processed; Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
2. The method for generating multimodal information based on a visual regression model according to claim 1, characterized in that: The image to be processed is processed based on the pre-trained target visual language model through the following steps: Inputting the image to be processed into a pre-trained multi-scale image encoder to obtain a multi-scale image embedding feature, wherein the multi-scale image embedding feature is used to characterize the image to be processed; Determine the image token to be processed by using the maximum distance similarity criterion in the multimodal vocabulary according to the multi-scale image embedding features; Two-dimensional position coding is used to positionally encode the image token to be processed; Input the to-be-processed image token and / or text token into a pre-trained VLM backbone network, adopt a strategy of predicting the next token of the target text and predicting the next-scale full token of the target image, and obtain a target text token and / or a target image multi-scale token; The multi-scale image token is input into a pre-trained multi-scale image decoder to obtain a target image.
3. The method for generating multimodal information based on a visual regression model according to claim 2, characterized in that: The steps of position encoding the multimodal target image token using two-dimensional position encoding include: In the first dimension position coding, the same coding is used for target image tokens of different scales of an image, and each token of text and special symbols occupies a separate position coding; In the second-dimensional position coding, the scale information of the target image token is encoded, and each sub-image token at the same scale has the same position coding, and text and special symbols use the same position coding.
4. The method for generating multimodal information based on a visual regression model according to claim 1, characterized in that: The following steps are used to process the text to be processed based on the pre-trained target visual language model: Determine a token of the text to be processed in the multimodal vocabulary according to the text to be processed; Input the to-be-processed text token and the to-be-processed image token into a pre-trained VLM backbone network, and adopt the next text token strategy to obtain a target text token; The target text tokens are converted according to the multimodal vocabulary to obtain the target text.
5. The method for generating multimodal information based on a visual regression model according to claim 1, characterized in that: The image to be processed is processed based on the pre-trained target visual language model through the following steps: Inputting the image to be processed into a multi-scale image encoder to obtain a multi-scale embedding feature of the image to be processed, using a maximum distance similarity criterion and a multimodal vocabulary to determine a token of the image to be processed; Input the to-be-processed image token and the to-be-processed text token into a pre-trained VLM backbone network, and adopt a strategy of predicting the next-scale full token of the target image to obtain the target image token; The target image token is converted according to the multimodal vocabulary to obtain a multi-scale embedded feature representation of the target image, which is input into a multi-scale image decoder to obtain the target image.
6. The method for generating multimodal information based on a visual regression model according to claim 1, characterized in that: The method further comprises: The number of tokens for each target image is pre-set; After the VLM backbone network outputs the target image start token, a full-token strategy for predicting the next scale of the target image is adopted. After the output of target image tokens at all scales is completed, the VLM backbone network automatically adds an image end token, where the number of scales and the number of tokens for each scale are pre-set based on the multi-scale image encoder and multi-scale image decoder.
7. A multimodal information generation device based on a visual regression model, characterized in that: The device comprises: An image and text acquisition module, used to acquire images and / or texts to be processed; An image and text generation module, used for inputting the image and / or text to be processed into a pre-trained target visual language model to obtain a target image and / or target text; Among them, the visual language model includes a multi-scale image encoder, a token encoding module, a VLM backbone network and a multi-scale decoder.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.