Visual information fusion method, device, equipment, medium and computer program product

Abstract

The invention provides a visual information fusion method, device and equipment, a medium and a computer program product, and the method comprises the steps: carrying out the coding of an input image and an input text, and obtaining a target mark sequence; the target marking sequence comprises a visual marking sequence and a language marking sequence; determining a fused visual context based on the attention of the language marker sequence to the visual marker sequence; based on the fused visual context, determining a modulation parameter of each target mark sequence; and based on the adjustment parameter, determining a semantic understanding result of the input text. According to the method, a dynamic feature modulation mechanism is introduced into each layer of the large language model, so that visual information can adaptively adjust text representation, and the ability of the large language model to understand multi-modal information is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a visual information fusion method, device, equipment, medium and computer program product. Background Art

[0002] Large language models (LLMs) are widely used in natural language processing (NLP). The core goal of combining multimodal learning and LLMs is to enable multimodal models to simultaneously understand information from different modalities, such as text, images, and videos. Currently, there are two main approaches for incorporating visual information into LLMs.

[0003] 1. Token-based splicing method: This method encodes image features into a series of visual tokens, which are then spliced ​​with text tokens and input into LLM for processing. However, due to the computational complexity of the self-attention mechanism of LLM, ,This method has high computational overhead when processing long sequences (such as video data). In addition, splicing a large number of visual tokens may interfere with the original language ability of LLM.

[0004] 2. Cross-Attention-Based Layer Insertion: This method introduces a cross-attention mechanism into each layer of the LLM, allowing text tokens to focus on visual tokens. While this method improves the utilization of visual information, it often requires a large number of additional parameters, and the added layers in the LLM structure may affect its original language modeling capabilities.

[0005] Therefore, how to effectively integrate visual information while maintaining the original structure and language capabilities of LLM while avoiding excessive computational overhead has become a technical problem that needs to be urgently solved in multimodal large language models. Summary of the Invention

[0006] The present invention provides a visual information fusion method, apparatus, device, medium and computer program product to address the defects of existing multimodal large language models in terms of structural and language capability changes when fusing visual information, and to realize the multimodal information understanding capability of the multimodal large language model when fusing visual information.

[0007] The present invention provides a visual information fusion method, comprising the following steps: Encoding the input image and the input text to obtain a target tag sequence; the target tag sequence includes a visual tag sequence and a language tag sequence; determining a fused visual context based on the attention of the language marker sequence to the visual marker sequence; determining a modulation parameter of each of the target marker sequences based on the fused visual context; Based on the adjustment parameters, a semantic understanding result of the input text is determined.

[0008] According to a visual information fusion method provided by the present invention, encoding the input image and input text to obtain a target tag sequence includes: Extract local features of the input image to obtain a high-dimensional feature map; Expanding the high-dimensional feature map in spatial dimensions to obtain an initial sequence; Alternatively, the input image is segmented to obtain image blocks; Performing linear projection mapping on the image blocks to obtain an initial sequence; Position encoding is performed on the initial sequence to obtain a visual mark sequence.

[0009] According to a visual information fusion method provided by the present invention, encoding the input image and input text to obtain a target tag sequence further includes: Cleaning the input text to obtain a cleaned text; Performing subword segmentation on the cleaned text to obtain a subword tag sequence; The subword tag sequence is sequentially subjected to index conversion, vector embedding, and position information injection to obtain a language tag sequence.

[0010] According to a visual information fusion method provided by the present invention, determining the fusion visual context based on the attention of the language marker sequence to the visual marker sequence includes: Obtaining an input representation vector; the input representation vector includes a query vector, a key vector, and a numerical vector; Determine a first vector based on the language tag sequence and the query vector; determine a second vector based on the visual tag sequence, the key vector and the numerical vector; The first vector and the second vector are input into a cross-attention module in a structural block of a large language model to obtain a fused visual context; the large language model is used to perform semantic understanding of the text.

[0011] According to a visual information fusion method provided by the present invention, determining the modulation parameters of each target marker sequence based on the fused visual context includes: determining visual information based on the fused visual context; The layer normalized position of the structure block is replaced by the visual information to obtain a modulation parameter of each target marker sequence.

[0012] According to a visual information fusion method provided by the present invention, the adjustment parameters include a scaling factor and a translation factor; and determining a semantic understanding result of the input text based on the adjustment parameters includes: Determine the mean and standard deviation of the language tag sequence at the current level; determining a language tag sequence of a next layer of the current layer based on the scaling factor, the mean, the standard deviation, and the translation factor; A semantic understanding result of the input text is determined based on the language tag sequence of the current layer and the language tag sequence of the next layer.

[0013] The present invention also provides a visual information fusion device, comprising the following modules: An encoding module, configured to encode an input image and an input text to obtain a target tag sequence; the target tag sequence includes a visual tag sequence and a language tag sequence; a fused visual context determining module, configured to determine a fused visual context based on the attention of the language marker sequence to the visual marker sequence; a modulation parameter determination module, configured to determine a modulation parameter of each target marker sequence based on the fused visual context; The text semantic understanding module is used to determine the semantic understanding result of the input text based on the adjustment parameters.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, any of the above-mentioned visual information fusion methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-described visual information fusion methods when executed by a processor.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned visual information fusion methods.

[0017] The visual information fusion method, apparatus, device, medium, and computer program product provided by the present invention encode an input image and input text to obtain a target tag sequence comprising a visual tag sequence and a language tag sequence. Based on the attention of the language tag sequence to the visual tag sequence, a fused visual context is determined. Based on the fused visual context, modulation parameters are determined for each target tag sequence. Ultimately, based on the adjusted parameters, a semantic understanding result of the input text is determined. By introducing a dynamic feature modulation mechanism at each layer of a large language model, the present invention enables visual information to adaptively adjust text representations, thereby enhancing the large language model's ability to understand multimodal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is one of the flow charts of the visual information fusion method provided by the present invention.

[0020] Figure 2 It is a structural diagram of a layer structure block of the large language model provided by the present invention.

[0021] Figure 3 This is the second flow chart of the visual information fusion method provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the visual information fusion device provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0025] The following combination Figure 1-Figure 5 The present invention describes the visual information fusion method, apparatus, device, medium and computer program product.

[0026] Figure 1This is one of the flow charts of the visual information fusion method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 100: Encode the input image and input text to obtain a target tag sequence; the target tag sequence includes a visual tag sequence and a language tag sequence; Specifically, if Figure 2 As shown in the figure, the various parts of a layer structure block of the large language model provided by the present invention are schematically shown as follows: Pointwise Feedforward: point-to-point feedforward; Scale, Shift: scaling, translation; Layer Norm: layer normalization; Multi-Head Self-Attention: multi-head self-attention; MLP: multi-layer perceptron; text token: text tag; Cross Attn: cross attention; vision token: visual tag.

[0027] Q, K, and V are the three input representation vectors used in the self-attention mechanism: Q represents the query vector; K represents the key vector; and V represents the value vector. These three vectors are derived from the original input vector through linear transformation. In the self-attention mechanism, based on Q, a weight distribution is calculated by calculating the similarity between Q and all K values. This weight distribution is then used to weight the sum of the associated V values. The concepts of Q, K, and V are derived from search systems. Q represents the query; K represents the key; and V represents the value. Q and K are matched for similarity, and after matching, V is weighted according to the similarity to produce the final output.

[0028] When the input is an image and text, the visual encoder extracts a sequence of image tokens; the text encoder extracts a sequence of text tokens.

[0029] Step 200: determining a fused visual context based on the attention of the language marker sequence to the visual marker sequence; Specifically, in the layer of the large language model, the cross-attention mechanism is used to adaptively integrate relevant visual condition information for each text tag to obtain a fused visual context.

[0030] Step 300: determining a modulation parameter of each target marker sequence based on the fused visual context; Specifically, at the layer normalization position of each structural block in the large language model, visual information is used to replace the original layer normalization and calculate the modulation parameters of each token. The modulation parameters are used to modulate the text token representation.

[0031] Step 400: Determine a semantic understanding result of the input text based on the adjustment parameters.

[0032] Specifically, during the computation of the large language model, feature modulation is performed on the intermediate representations. During training and inference, multimodal alignment and multimodal instruction fine-tuning are performed on multimodal data, for tasks such as image conversation, video conversation, image question answering, video question answering, image captioning, and video captioning, enabling the large language model to learn appropriate visual modulation parameters. During inference, the large language model adaptively adjusts the language representation based on the input visual content, generating text output that better matches the visual context, ultimately achieving a semantic understanding of the input text (the textual answer).

[0033] This embodiment encodes the input image and input text to obtain a target tag sequence consisting of a visual tag sequence and a linguistic tag sequence. Based on the attention of the linguistic tag sequence to the visual tag sequence, a fused visual context is determined. Based on the fused visual context, modulation parameters are determined for each target tag sequence. Ultimately, based on the adjustment parameters, the information contained in the visual tags is injected into the encoding process of the text tags. By introducing a dynamic feature modulation mechanism at each layer of the large language model, the present invention enables visual information to adaptively adjust text representations, thereby enhancing the large language model's ability to understand multimodal information.

[0034] In one embodiment, the visual information fusion method provided by the embodiment of the present invention may further include: Step 110: extract local features from the input image to obtain a high-dimensional feature map; Expanding the high-dimensional feature map in spatial dimensions to obtain an initial sequence; Alternatively, in step 120, the input image is segmented to obtain image blocks; Performing linear projection mapping on the image blocks to obtain an initial sequence; Step 130: Perform position encoding on the initial sequence to obtain a visual marker sequence.

[0035] In the visual feature encoding part, a pre-trained visual model (such as ViT, ResNet, and CLIP) is used to encode the input image or video frame to obtain a visual token sequence. , which is the visual marking sequence in this embodiment.

[0036] Specifically, the process of converting the input image into a visual token sequence by the visual model includes the following core steps: 1. Image segmentation and feature extraction: ‌ResNet-like CNN model: extracts local features of the input image layer by layer through convolutional layers to generate high-dimensional feature maps, gradually compressing the spatial dimensions and increasing the number of channels; ‌ViT / CLIP-like Transformer model‌: The input image is divided into fixed-size 2D image blocks, each of which is mapped to a vector through linear projection to form an initial visual token sequence‌.

[0037] 2. Positional Encoding Fusion: To preserve the spatial information of the image, positional encoding needs to be added to the token sequence. For example, ViT performs a learnable 1D or 2D encoding on the position of each image block, adds it to the image block vector, and then inputs it into the Transformer.

[0038] 3. Sequencing and context modeling: Through stacked Transformer encoder layers, the self-attention mechanism is used to model the global dependencies between tokens, and ultimately outputs a visual token sequence containing semantic information.

[0039] This embodiment obtains visual information to be fused by encoding the input image.

[0040] In one embodiment, the visual information fusion method provided by the embodiment of the present invention may further include: Step 140: Clean the input text to obtain a cleaned text; Step 150: performing subword segmentation on the cleaned text to obtain a subword tag sequence; Step 160: Perform index conversion, vector embedding, and position information injection on the subword tag sequence in sequence to obtain a language tag sequence.

[0041] In the language (text) feature encoding part, the model encodes the input text to obtain a text token sequence.

[0042] The process generally includes the following steps: 1. Preprocessing and word segmentation 1. Text cleaning: Remove special characters and extra spaces; unify the format; some models remove stop words to reduce noise. This yields the cleaned text used in this example.

[0043] 2. Word segmentation strategy (including subword segmentation and special tag addition) Subword segmentation: Split the text into subwords or word-level tokens through a word segmenter, that is, a subword token sequence in this embodiment.

[0044] Special mark addition: insert start and delimiter characters, etc., to identify text boundaries or task types.

[0045] 2. Digital Mapping and Embedding 1. Index conversion: Map the token to a numeric ID in the vocabulary. Each token corresponds to a unique index in the vocabulary.

[0046] 2. Vector Embedding: The embedding layer converts the numerical index into a high-dimensional vector, and the position in the vector space reflects the semantic similarity of the token.

[0047] ‌Static embedding‌: pre-trained fixed vectors; ‌Dynamic embedding‌: dynamically adjust vectors based on context. ‌

[0048] 3. Positional Encoding and Context Modeling 1. Position information injection: To preserve the sequence order, position encoding is added to the Token vector. Transformer uses a sine function or learnable parameters to generate position encoding.

[0049] 2. Attention mechanism processing: Calculate the dependencies between tokens through self-attention to generate contextual representations containing global semantics.

[0050] 4. Output and Optimization 1. Task adaptation (including classification tasks and generation tasks) Classification task: Use the vector of the start token as the full sentence representation to input the classifier.

[0051] ‌Generation Task‌: Generate a token sequence step by step through the decoder‌.

[0052] 2. Feedback optimization: During the training process, the embedding and model parameters are adjusted through the loss function to improve the semantic discrimination of the token representation.

[0053] This embodiment encodes the input text to obtain the language tag sequence to be fused.

[0054] In one embodiment, the visual information fusion method provided by the embodiment of the present invention may further include: Step 210: Obtain an input representation vector; the input representation vector includes a query vector, a key vector, and a numerical vector; Step 220: Determine a first vector based on the language tag sequence and the query vector; determine a second vector based on the visual tag sequence, the key vector, and the numerical vector; Step 230: Input the first vector and the second vector into a cross-attention module in a structural block of a large language model to obtain a fused visual context; the large language model is used to perform semantic understanding of the text.

[0055] Specifically, in the part where visual information interacts with language tag sequences, such as Figure 2As shown, in some or all layers of LLM, a cross-modal attention mechanism is introduced to calculate the attention of the language tag sequence to the visual tag sequence, as shown in Formula 1, where The text tag sequence for the current layer; , and is the projection matrix; Represents the fused visual context.

[0056] ; (1) This embodiment calculates the fused visual context by introducing a cross-modal attention mechanism into the layers of a large language model.

[0057] In one embodiment, the visual information fusion method provided by the embodiment of the present invention may further include: Step 310: determining visual information based on the fused visual context; Step 320: Replace the layer-normalized position of the structural block with the visual information to obtain a modulation parameter of each target marker sequence.

[0058] Specifically, in the feature modulation-based visual fusion part, such as Figure 2 As shown in the LayerNorm position of each Block (structural block) of LLM, the original LayerNorm is replaced by visual information to calculate the modulation parameters of each Token. As shown in Formula 2, is the scaling factor; is the translation factor used to modulate the text Token representation.

[0059] ; (2) This embodiment replaces the layer normalization position of the large language model with visual information to obtain the Token modulation parameters.

[0060] Figure 2 This is the second flow chart of the visual information fusion method provided by the present invention, such as Figure 2 As shown, the method may further include: Step 410: Determine the mean and standard deviation of the language tag sequence of the current layer; Step 420: Determine a language tag sequence of a next layer of the current layer based on the scaling factor, the mean, the standard deviation, and the translation factor. Step 430: Determine a semantic understanding result of the input text based on the language tag sequence of the current layer and the language tag sequence of the next layer.

[0061] Specifically, the feature modulation-based visual fusion part further includes: performing feature modulation as shown in Formula 3 on the intermediate representation during the LLM calculation process. is the mean; is the standard deviation; Control scale, Controls the offset.

[0062] ; (3) The visual information fusion method provided by the present invention only injects visual information into a part of the structural blocks of a large language model, which can achieve a balance between efficiency and accuracy.

[0063] This embodiment improves the ability of the large language model to understand multimodal information by only requiring feature modulation in the intermediate representation process of the large language model.

[0064] The visual information fusion device provided by the present invention is described below. The visual information fusion device described below and the visual information fusion method described above can be referenced to each other.

[0065] Please refer to Figure 4 The present invention also provides a visual information fusion device, comprising: The encoding module 401 is used to encode the input image and the input text to obtain a target tag sequence; the target tag sequence includes a visual tag sequence and a language tag sequence; a fused visual context determining module 402 for determining a fused visual context based on the attention of the language marker sequence to the visual marker sequence; A modulation parameter determination module 403 is configured to determine a modulation parameter of each target marker sequence based on the fused visual context; The text semantic understanding module 404 is configured to determine a semantic understanding result of the input text based on the adjustment parameters.

[0066] Optionally, the encoding module includes: A local feature extraction unit is used to extract local features of the input image to obtain a high-dimensional feature map; A spatial dimension expansion unit, configured to perform spatial dimension expansion on the high-dimensional feature map to obtain an initial sequence; An image segmentation unit, configured to segment an input image into image blocks; a linear projection mapping unit, configured to perform linear projection mapping on the image block to obtain an initial sequence; A position encoding unit is used to perform position encoding on the initial sequence to obtain a visual mark sequence.

[0067] Optionally, the encoding module further includes: A text cleaning unit, configured to clean the input text to obtain a cleaned text; A subword segmentation unit, configured to perform subword segmentation on the cleaned text to obtain a subword tag sequence; The language tag sequence determination unit is used to perform index conversion, vector embedding and position information injection on the subword tag sequence in sequence to obtain a language tag sequence.

[0068] Optionally, the fused visual context determination module includes: An input representation vector acquisition unit, configured to acquire an input representation vector; the input representation vector includes a query vector, a key vector, and a numerical vector; an input vector determining unit, configured to determine a first vector based on the language tag sequence and the query vector; and determine a second vector based on the visual tag sequence, the key vector, and the numerical vector; The fused visual context determining unit is configured to input the first vector and the second vector into a cross-attention module in a structural block of a large language model to obtain a fused visual context; the large language model is used to perform semantic understanding of the text.

[0069] Optionally, the modulation parameter determination module includes: a visual information determining unit, configured to determine visual information based on the fused visual context; The visual information replacement unit is configured to replace the layer-normalized position of the structural block with the visual information to obtain a modulation parameter of each target marker sequence.

[0070] Optionally, the text semantic understanding module includes: a value calculation unit for determining the mean and standard deviation of the language tag sequence of the current layer; a lower-layer language marker sequence determining unit, configured to determine a language marker sequence of a layer below the current layer based on the scaling factor, the mean, the standard deviation, and the translation factor; The semantic understanding unit is used to determine the semantic understanding result of the input text based on the language tag sequence of the current layer and the language tag sequence of the next layer.

[0071] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communications bus 540. The processor 510 may call logic instructions in the memory 530 to execute a visual information fusion method, which includes: encoding an input image and an input text to obtain a target tag sequence; the target tag sequence includes a visual tag sequence and a language tag sequence; determining a fused visual context based on the attention of the language tag sequence to the visual tag sequence; determining a modulation parameter for each target tag sequence based on the fused visual context; and determining a semantic understanding result of the input text based on the adjustment parameter.

[0072] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the visual information fusion method provided by the above methods, which includes: encoding the input image and input text to obtain a target marker sequence; the target marker sequence includes a visual marker sequence and a language marker sequence; determining a fused visual context based on the attention of the language marker sequence to the visual marker sequence; determining the modulation parameters of each target marker sequence based on the fused visual context; and determining the semantic understanding result of the input text based on the adjustment parameters.

[0074] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the visual information fusion method provided by the above-mentioned methods, the method comprising: encoding an input image and an input text to obtain a target marker sequence; the target marker sequence comprises a visual marker sequence and a language marker sequence; determining a fused visual context based on the attention of the language marker sequence to the visual marker sequence; determining modulation parameters of each of the target marker sequences based on the fused visual context; and determining a semantic understanding result of the input text based on the adjustment parameters.

[0075] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0076] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A visual information fusion method, characterized in that: include: Encoding the input image and the input text to obtain a target tag sequence; the target tag sequence includes a visual tag sequence and a language tag sequence; determining a fused visual context based on the attention of the language marker sequence to the visual marker sequence; determining a modulation parameter of each of the target marker sequences based on the fused visual context; Based on the adjustment parameters, a semantic understanding result of the input text is determined.

2. The visual information fusion method according to claim 1, characterized in that: The encoding of the input image and the input text to obtain the target tag sequence includes: Extract local features of the input image to obtain a high-dimensional feature map; Expanding the high-dimensional feature map in spatial dimensions to obtain an initial sequence; Alternatively, the input image is segmented to obtain image blocks; Performing linear projection mapping on the image blocks to obtain an initial sequence; Position encoding is performed on the initial sequence to obtain a visual mark sequence.

3. The visual information fusion method according to claim 1, characterized in that: The encoding of the input image and the input text to obtain a target tag sequence further includes: Cleaning the input text to obtain a cleaned text; Performing subword segmentation on the cleaned text to obtain a subword tag sequence; The subword tag sequence is sequentially subjected to index conversion, vector embedding, and position information injection to obtain a language tag sequence.

4. The visual information fusion method according to claim 1, characterized in that: The determining of the fused visual context based on the attention of the language marker sequence to the visual marker sequence comprises: Obtaining an input representation vector; the input representation vector includes a query vector, a key vector, and a numerical vector; Determine a first vector based on the language tag sequence and the query vector; determine a second vector based on the visual tag sequence, the key vector and the numerical vector; The first vector and the second vector are input into a cross-attention module in a structural block of a large language model to obtain a fused visual context; the large language model is used to perform semantic understanding of the text.

5. The visual information fusion method according to claim 4, characterized in that: The determining, based on the fused visual context, a modulation parameter of each target marker sequence includes: determining visual information based on the fused visual context; The layer normalized position of the structure block is replaced by the visual information to obtain a modulation parameter of each target marker sequence.

6. The visual information fusion method according to claim 1, characterized in that: The adjustment parameters include a scaling factor and a translation factor; Determining a semantic understanding result of the input text based on the adjustment parameter includes: Determine the mean and standard deviation of the language tag sequence at the current level; determining a language tag sequence of a next layer of the current layer based on the scaling factor, the mean, the standard deviation, and the translation factor; A semantic understanding result of the input text is determined based on the language tag sequence of the current layer and the language tag sequence of the next layer.

7. A visual information fusion device, characterized in that: include: An encoding module, configured to encode an input image and an input text to obtain a target tag sequence; the target tag sequence includes a visual tag sequence and a language tag sequence; a fused visual context determining module, configured to determine a fused visual context based on the attention of the language marker sequence to the visual marker sequence; a modulation parameter determination module, configured to determine a modulation parameter of each target marker sequence based on the fused visual context; The text semantic understanding module is used to determine the semantic understanding result of the input text based on the adjustment parameters.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the visual information fusion method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the visual information fusion method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the visual information fusion method according to any one of claims 1 to 6 is implemented.

Images