Interaction method based on multi-modal large model, storage medium and electronic device

Abstract

The invention discloses an interaction method based on a multi-modal large model, a storage medium and an electronic device, and relates to the technical field of smart home, and the interaction method based on the multi-modal large model comprises the steps: carrying out the deep semantic analysis of a user instruction through a routing network, and generating a routing vector group; the routing vector group comprises an encoder control vector and a fusion device control vector; controlling a self-attention mechanism of each modal encoder in at least one modal encoder in the vector modulation multi-modal large model through an encoder, and generating modal characteristics of target modal information input into each modal encoder; and controlling the multi-modal large model to fuse the modal features of the multi-modal information corresponding to the user instruction through the fusion device control vector, and generating interaction information corresponding to the user instruction. The problem that a multi-modal large model cannot dynamically process multi-modal information based on a user instruction is solved. The technical effect that processing of the multi-modal information by the multi-modal large model is strongly related to the user instruction is achieved.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to an interaction method, storage medium, and electronic device based on a multimodal large model. Background Technology

[0002] In related technologies, multimodal large models face the following technical bottlenecks when processing user instructions: 1) Static and coarse-grained fusion: Multimodal large models typically employ a fixed fusion architecture (such as early fusion, mid-term fusion, or late fusion). Regardless of the task indicated by the user instruction, all information from all modalities interacts in the same way. For example, when processing the task of "describe this picture," the multimodal model analyzes all the details of the entire picture; however, when processing a specific question like "how many cats are in the picture?", the multimodal model still activates visual regions unrelated to the task (such as background textures), resulting in significant computational redundancy. 2) Modal interference and noise: In multimodal information, not all modalities are equally important to the current task. For example, when answering a video-based "what happened?" question, visual information may dominate, while audio may be background music, which is noise. Multimodal large models treat all modalities equally, which can easily lead to secondary or noisy modalities interfering with the judgment of the primary modality, reducing inference accuracy. 3) Task Intent Understanding Bias: The understanding of user instructions by the large multimodal model is relatively decoupled from the subsequent multimodal information processing flow. The large model may understand the user instructions, but its internal components such as the visual encoder and fusion unit are not precisely "guided" to focus on the information most relevant to the user instructions, resulting in a mismatch between intent and perception.

[0003] In other words, the relevant technologies suffer from problems such as a fixed multimodal large model fusion architecture, equal treatment of all modal information, and relative decoupling of task understanding and multimodal information processing, which prevent the multimodal large model from dynamically processing multimodal information based on user instructions.

[0004] There is still no effective solution to the problem that large multimodal models cannot dynamically process multimodal information based on user commands in related technologies.

[0005] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0006] This application provides an interaction method, storage medium, and electronic device based on a multimodal large model, to at least solve the problem in related technologies that multimodal large models cannot dynamically process multimodal information based on user instructions.

[0007] According to one aspect of the embodiments of this application, an interaction method based on a multimodal large model is provided, comprising: performing deep semantic parsing on user instructions through a routing network to generate a routing vector group; wherein the routing vector group includes: an encoder control vector and a fusion control vector, the encoder control vector being used to control at least one modal encoder in the multimodal large model; modulating the self-attention mechanism of each modal encoder in the at least one modal encoder through the encoder control vector to generate modal features of target modal information input to each modal encoder; controlling the multimodal large model to fuse the modal features of the multimodal information corresponding to the user instruction through the fusion control vector to generate interaction information corresponding to the user instruction; wherein the modal features of the multimodal information include: the modal features of the target modal information.

[0008] In an exemplary embodiment, a routing network performs deep semantic parsing on user instructions to generate a routing vector group, including: converting the user instructions into target vectors using a text encoder and a multilayer perceptron; slicing the target vectors based on a preset dimension using a vector decomposition layer to obtain multiple slice vectors; and processing each slice vector according to the activation function corresponding to each slice vector to obtain the routing vector group; wherein the routing network includes: the text encoder, a multilayer perceptron connected to the text encoder, and a vector decomposition layer connected to the multilayer perceptron.

[0009] In an exemplary embodiment, modal features of target modal information input to each modal encoder are generated by modulating the self-attention mechanism of each modal encoder in the at least one modal encoder through the encoder control vector, including: modulating the query matrix and key matrix of the self-attention mechanism respectively through the encoder control vector; and determining the modal features of the target modal information through the modulated query matrix and the modulated key matrix.

[0010] In an exemplary embodiment, the query matrix and key matrix of the self-attention mechanism are modulated by the encoder control vector, respectively, including: determining the modulated query matrix and the modulated key matrix by the spatial routing vector and the semantic routing vector in the encoder control vector, respectively; wherein, the spatial routing vector is used to indicate the position information that the user instruction requires attention in the target modality information, and the semantic routing vector is used to indicate the attribute information that the user instruction requires attention in the target modality information.

[0011] In an exemplary embodiment, modulating the query matrix and key matrix of the self-attention mechanism using the encoder control vector includes: when the encoder control vector includes a spatial routing vector and a semantic routing vector, determining the modulated query matrix using the following formula: The modulated bond matrix is ​​determined using the following formula: Wherein, Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; and For learnable projection matrices, This indicates element-wise multiplication; For spatial routing vectors, This is a semantic routing vector.

[0012] In an exemplary embodiment, determining the modal features of the target modal information using a modulated query matrix and a modulated key matrix includes: determining the modal features of the target modal information using the following formula, including: Where Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; ConditionalAttention(Q,K,V|r) represents the modal features of the target modal information, and V represents the value matrix obtained after linear transformation of the target modal information. This represents the dimension of the key vectors in the key matrix.

[0013] In an exemplary embodiment, modal features of target modal information input to each modal encoder are generated by modulating the self-attention mechanism of each modal encoder using the encoder control vector. This includes: modulating the query matrix of the self-attention mechanism using the spatial routing vector and a first projection bias term when the encoder control vector includes a spatial routing vector and a semantic routing vector; and modulating the key matrix of the self-attention mechanism using the semantic routing vector and a second projection bias term. The spatial routing vector indicates the location information that the user instruction requires attention in the target modal information, and the semantic routing vector indicates the attribute information that the user instruction requires attention in the target modal information. The modal features of the target modal information are determined using an attention mask matrix, the modulated query matrix, and the modulated key matrix.

[0014] In an exemplary embodiment, the modal features of the target modal information are determined using an attention mask matrix, a modulated query matrix, and a modulated key matrix, including: Where Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; ConditionalAttention(Q,K,V|r) represents the modal features of the target modal information, and V represents the value matrix obtained after linear transformation of the target modal information. The dimension of the key vectors in the key matrix is ​​represented by , and M represents the attention mask matrix.

[0015] In an exemplary embodiment, the fusion control vector controls the multimodal large model to fuse the modal features of the multimodal information corresponding to the user instruction, generating interactive information corresponding to the user instruction. This includes: modal gating the modal features of the multimodal information using a modal weight vector to obtain gated features; and fusing the gated features using a fusion control vector to generate interactive information corresponding to the user instruction. The fusion control vector includes the modal weight vector and the fusion control vector. The modal weight vector indicates the relative importance of each modal information to the user instruction, and the fusion control vector indicates the allowed fusion strategies for the multimodal information.

[0016] In an exemplary embodiment, modal gating is performed on the modal features of the multimodal information using modal weight vectors to obtain gated features, including determining the gated features using the following formula: ;in, Indicates the features after gating. Modal features representing each modal information For the sigmoid function, , Let be the modal weight vector.

[0017] In an exemplary embodiment, the interaction information corresponding to the user instruction is generated by fusing gated features through a fusion control vector, including: determining the fusion strategy weight vector corresponding to the fusion control vector through a dynamic fusion processor; fusing the gated features through the fusion strategy weight vector to obtain a fusion vector; and generating the interaction information based on the fusion vector using the multimodal large model.

[0018] In an exemplary embodiment, determining the fusion strategy weight vector corresponding to the fusion control vector through a dynamic fusion processor includes: determining the fusion strategy weight vector using the following formula, including: ;in, This represents the weight vector of the fusion strategy. This represents the fusion control vector, which is used to specify the fusion strategy.

[0019] In an exemplary embodiment, the gated features are fused using the fusion strategy weight vector to obtain a fusion vector, including: determining the fusion vector using the following formula, including: ;in, Represents the fusion vector. For the j-th predefined fusion strategy, The number of modes in multimodal information; This indicates the number of predefined fusion strategies. This represents the weight of the j-th fusion strategy. and .

[0020] According to another aspect of the embodiments of this application, an interactive device based on a multimodal large model is also provided, comprising: a parsing module, configured to perform deep semantic parsing on user instructions through a routing network to generate a routing vector group; wherein the routing vector group includes: an encoder control vector and a fusion control vector, the encoder control vector being used to control at least one modal encoder in the multimodal large model; a modulation module, configured to modulate the self-attention mechanism of each modal encoder in the at least one modal encoder through the encoder control vector to generate modal features of target modal information input to each modal encoder; and a control module, configured to control the multimodal large model to fuse the modal features of the multimodal information corresponding to the user instruction through the fusion control vector to generate interactive information corresponding to the user instruction; wherein the modal features of the multimodal information include: the modal features of the target modal information.

[0021] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described interaction method based on a multimodal large model at runtime.

[0022] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described interaction method based on a multimodal large model through the computer program.

[0023] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described interaction method based on a multimodal large model.

[0024] This application utilizes a routing network to perform deep semantic parsing of user commands, generating a routing vector set. This routing vector set includes an encoder control vector and a fusion control vector. The encoder control vector controls at least one modal encoder in a multimodal large model. The encoder control vector modulates the self-attention mechanism of each modal encoder, generating modal features of the target modal information input to each modal encoder. The fusion control vector controls the multimodal large model to fuse the modal features of the multimodal information corresponding to the user command, generating interactive information corresponding to the user command. The modal features of the multimodal information include the modal features of the target modal information. This ensures that the generation and fusion of modal features in the multimodal large model's processing of multimodal information are controlled by the routing vector set obtained from deep semantic parsing of the user command. Therefore, this technical solution solves the problem in related technologies where multimodal large models cannot dynamically process multimodal information based on user commands. It achieves the technical effect of strongly correlating the processing of multimodal information by the multimodal large model with user commands. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the hardware environment for an optional interaction method based on a multimodal large model according to an embodiment of this application;

[0028] Figure 2 This is a flowchart of an optional interaction method based on a multimodal large model according to an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of the system architecture according to an embodiment of this application;

[0030] Figure 4 This is another flowchart of an optional interaction method based on a multimodal large model according to an embodiment of this application;

[0031] Figure 5 This is yet another flowchart of an optional interaction method based on a multimodal large model according to an embodiment of this application;

[0032] Figure 6 This is a structural block diagram of an optional interactive device based on a multimodal large model according to an embodiment of this application. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] According to one aspect of the embodiments of this application, an interaction method for smart home devices is provided. This interaction method for smart home devices is widely applicable to digital control scenarios such as smart homes, smart home ecosystems, intelligence house ecosystems, whole-house intelligence, digital homes, networked homes, automated homes, electronic homes, digital homes, and digital homes. It relies on technologies such as the Internet of Things (IoT), cloud computing, artificial intelligence (AI), digital technology, digital twins, and AR / VR to construct an intelligent, automated, personalized, and customized digital ecosystem. Optionally, in this embodiment, the above-mentioned interaction method for smart home devices can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0036] The aforementioned network communication architecture includes, but is not limited to, two types of links: wired transmission and wireless connection. Among them, the wired network can cover wide area networks, metropolitan area networks, and local area networks, while the wireless network supports communication standards such as WIFI, Bluetooth, and 5G. It can adapt to the digital technology transmission needs of the Internet of Things (IoT), the Internet of Things, and the Internet of Things blockchain, ensuring stable interaction of digital information, enabling devices to make autonomous, accurate, precise, and accurate judgments and adjustments, and supporting personalized and customized intelligent capabilities such as voice interaction, remote control, and self-control. Terminal devices 102 include, but are not limited to, PCs, mobile phones, tablets, and all categories of digital home appliances, digital electrical appliances, smart home appliances, and whole-house smart devices. Specifically, they include: smart air conditioners, air conditioning equipment, smart range hoods, smart refrigerators, smart ovens, grilling and cooking equipment, smart stoves, smart washing machines, washing, rinsing, drying and garment processing equipment, smart water heaters, smart washing equipment, smart dishwashers, smart projectors, smart TVs, smart clothes racks, smart curtains, smart audio-visual equipment, smart sockets, smart speakers, smart speakers, smart fresh air systems, smart kitchen and bathroom systems, kitchens, cabinets, kitchen cabinet equipment, smart bathroom equipment, robot vacuum cleaners, robot window cleaners, robot mopping, air purifiers, smart steam ovens, smart microwave ovens, smart water heaters, smart air purifiers, smart water dispensers, smart door locks, smart lighting fixtures, smart light bulbs, smart air detectors, smart remote controls, and smart gateways. They also cover smart terminals for security, lighting, and other digital home furnishings.

[0037] This embodiment provides an interaction method based on a multimodal large model, including but not limited to applications in terminal devices. Figure 2 This is a flowchart of an optional interaction method based on a multimodal large model according to an embodiment of this application. The process includes the following steps:

[0038] Step S202: Perform deep semantic parsing of user commands through a routing network to generate a routing vector group; wherein, the routing vector group includes: encoder control vector and fusion control vector, the encoder control vector being used to control at least one modal encoder in the multimodal large model;

[0039] The routing network can reuse several Transformer encoding layers (such as layers 1-4) from the front end of the multimodal large model. Alternatively, it can employ a separately deployed, dedicated lightweight text encoder, such as TinyBERT, DistilBERT, or a 3-layer Transformer. This lightweight text encoder is connected to the multimodal large model. The routing network is responsible for performing deep semantic parsing of user instructions to generate a set of routing vectors that guide the subsequent encoding and fusion processes of the multimodal large model. User instructions are natural language instructions, such as "What is the red object in the upper left corner of the image?"

[0040] Step S204: Modal features of the target modal information input to each modal encoder are generated by modulating the self-attention mechanism of each modal encoder in the at least one modal encoder through the encoder control vector.

[0041] The encoder includes at least one of the following: a visual encoder, a speech encoder, and a text encoder. The encoder control vector is injected into the self-attention calculation of each corresponding modal encoder, performing element-wise modulation on the query matrix Query(Q) and the key matrix Key(K). This ensures that each modal encoder focuses only on the specific region or feature indicated by the user command, such as only calculating attention for the "top left corner" and "red" regions, ignoring irrelevant regions, thereby generating modal features of the target modal information strongly correlated with the user command. Optionally, the target modal information of the visual encoder is visual modal information, the target modal information of the speech encoder is speech modal information, the target modal information of the text encoder is text modal information, and so on.

[0042] Step S206: Control the multimodal large model to fuse the modal features of the multimodal information corresponding to the user command through the fusion control vector, and generate the interaction information corresponding to the user command; wherein, the modal features of the multimodal information include: the modal features of the target modal information.

[0043] By controlling the fusion vector, the modal weights of the modal features of multimodal information and the allowed fusion methods can be determined during the fusion process, thereby achieving dynamic and fine-grained modal feature fusion. Furthermore, the target modal information is the modality explicitly pointed to in the user command (such as image or audio), while multimodal information may contain multiple modalities (such as image + text + speech).

[0044] Through the above steps, the routing vector group corresponding to the user command is parsed out through the routing network; the self-attention mechanism of the encoder of the multimodal large model is modulated by the first vector in the routing vector group to obtain conditional self-attention; the modal features of the target modal information input to the encoder are determined by the conditional self-attention; the second vector in the routing vector group controls the multimodal large model to fuse the modal features of the multimodal information corresponding to the user command, generating the interaction information corresponding to the user command; wherein, the modal features of the multimodal information include: the modal features of the target modal information; the first vector and the second vector are vectors of different vector types. This ensures that the multimodal large model is controlled by the routing vector group corresponding to the user command in both the modal feature extraction and feature fusion stages. Therefore, by adopting the above technical solution, the problem that multimodal large models cannot dynamically process multimodal information based on user commands in related technologies is solved. This achieves the technical effect of strongly correlating the processing of multimodal information by the multimodal large model with user commands.

[0045] In an exemplary embodiment, a routing network performs deep semantic parsing on user instructions to generate a routing vector group, including: converting the user instructions into target vectors using a text encoder and a multilayer perceptron; slicing the target vectors based on a preset dimension using a vector decomposition layer to obtain multiple slice vectors; and processing each slice vector according to the activation function corresponding to each slice vector to obtain the routing vector group; wherein the routing network includes: the text encoder, a multilayer perceptron connected to the text encoder, and a vector decomposition layer connected to the multilayer perceptron.

[0046] The process of converting the user instruction into a target vector using a text encoder and a multilayer perceptron includes: performing deep semantic parsing on the user instruction using a text encoder to obtain a text feature representation corresponding to the user instruction; and converting the text feature representation into the target vector using a multilayer perceptron. The preset dimension is the vector dimension of each vector in the routing vector group.

[0047] The aforementioned routing network, officially named Task-Aware Dynamic Routing Network, is a lightweight controller that generates a set of structured routing vectors based on the user's task intent. This dynamically guides the perception, fusion, and reasoning processes of a large multimodal model, achieving a paradigm shift from "static fusion" to "dynamic routing" and defining an information processing pipeline based on task intent.

[0048] Specifically, the architecture of the routing network can include: an input layer for receiving user instructions in text form (or a task description of the user instructions); a text encoder connected to the input layer, comprising 4 Transformer layers with a hidden dimension of 512 and 8-head attention, used to process the user instructions into text feature representations; a routing vector generation MLP (Multi-Layer Perceptron) connected to the text encoder, specifically comprising 3 MLP layers [512→256→128→N], using the GELU (Gaussian Error Linear Unit) activation function, used to transform the text feature representation into the aforementioned target vector; a routing vector decomposition layer (equivalent to the aforementioned vector decomposition layer) connected to the routing vector generation MLP, used to decompose the aforementioned target vector into multiple independent slice vectors based on a preset dimension; and an output layer connected to the routing vector decomposition layer, used to process each slice vector in the multiple slice vectors through a corresponding activation function and output a group of routing vectors. Optionally, the network parameter configuration of this routing network is shown in Table 1.

[0049] Table 1

[0050]

[0051] Optionally, the core algorithm of the routing network is: ;

[0052] ;

[0053] ; ;

[0054] ; ;

[0055] in, H represents the user instruction, H represents the text feature representation, and V represents the target vector mentioned above.

[0056] The routing vector set output after the above routing network processing specifically includes: spatial routing vector ( ): The preset dimension is 196 (14×14), used to indicate the spatial attention distribution related to the encoding task; the generation method is MLP+Softmax normalization, that is, the activation function used to process each slice vector corresponding to the spatial routing vector is Softmax. Semantic routing vector ( ): The preset dimension is 1024, used to indicate semantic concepts related to the encoding task; the generation method is MLP+Tanh activation, that is, the activation function used to process each slice vector corresponding to the semantic routing vector is Tanh. Modal weight vector ( ): Preset dimension is ( (A positive integer representing the number of modalities in the multimodal information), used to indicate the importance of each encoded modality; the generation method is MLP+Softmax normalization, that is, the activation function used to process each slice vector corresponding to the modality weight vector is Softmax. Fusion control vector ( ): The preset dimension is 256, which is used to specify the fusion strategy; the generation method is: MLP (no specific activation), that is, each slice vector corresponding to the fusion control vector is processed without specific activation.

[0057] For example, the process of generating a route vector group includes the following steps 1 to 5:

[0058] Step 1: Task Analysis. Input: "What is the red object in the upper left corner of the image?"; Output: Text feature representation.

[0059] Step 2: Route Vector Generation. Generate four route vectors: the spatial route vector indicates focus on the top left corner; the semantic route vector indicates focus on the red object; and the modal weight vector indicates that the visual modal information has a weight of 0.9.

[0060] Step 3: Dynamic visual encoding. The route vector modulated ViT (Vision Transformer, visual encoder) attention makes the visual encoder focus only on the relevant image region (top left corner, red object region).

[0061] Step 4: Dynamic Fusion. Select a fusion strategy based on the fusion control vector, such as choosing a question-and-answer interaction fusion strategy.

[0062] Step 5: Final inference: LLM (Large Language Model) generates interactive information: "This is a red apple." Here, LLM refers to deep learning models with millions or even billions of parameters.

[0063] In an exemplary embodiment, modal features of target modal information input to each modal encoder are generated by modulating the self-attention mechanism of each modal encoder in the at least one modal encoder through the encoder control vector, including: modulating the query matrix and key matrix of the self-attention mechanism respectively through the encoder control vector; and determining the modal features of the target modal information through the modulated query matrix and the modulated key matrix.

[0064] Furthermore, the query matrix and key matrix of the self-attention mechanism are modulated by the encoder control vector, including: determining the modulated query matrix and modulated key matrix by the spatial routing vector and semantic routing vector in the encoder control vector, respectively; wherein, the spatial routing vector is used to indicate the position information that the user instruction requires attention in the target modality information, and the semantic routing vector is used to indicate the attribute information that the user instruction requires attention in the target modality information.

[0065] Regarding the location information indicated by the spatial routing vector: for the visual encoder, it is spatial location information, such as "top left corner of the image"; for the text encoder and speech encoder, it is the length location information that needs attention, such as the audio length location and the text length location that needs attention. Regarding the attribute information indicated by the semantic routing vector, for example, if the user instruction is "What is the red object in the top left corner of the image?", the attribute information indicated by the parsed semantic routing vector would be: red, object.

[0066] The modulated query matrix and modulated key matrix are determined using the spatial routing vector and semantic routing vector in the encoder control vector, respectively. This includes: multiplying the spatial routing vector element-wise with the query matrix to obtain the modulated query matrix; and multiplying the semantic routing vector element-wise with the key matrix to obtain the modulated key matrix. That is, the modulated query matrix can be determined using the following formula: The modulated bond matrix is ​​determined using the following formula: Wherein, Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; and For learnable projection matrices, This indicates element-wise multiplication; For spatial routing vectors, This is a semantic routing vector.

[0067] Specifically, determining the modal features of the target modal information using the modulated query matrix and the modulated key matrix includes: determining the modal features of the target modal information using the following formula, including: Where Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; ConditionalAttention(Q,K,V|r) represents the modal features of the target modal information, and V represents the value matrix obtained after linear transformation of the target modal information. This represents the dimension of the key vectors in the key matrix.

[0068] By modulating the self-attention mechanism, the model's focus on task-relevant regions is significantly enhanced, thereby more effectively filtering out irrelevant information and improving processing efficiency and accuracy. Spatial routing vectors guide the model to focus on specific locations in an image or scene, while semantic routing vectors enhance the understanding of instruction keywords or concepts. The combination of the two enables the model's attention mechanism to intelligently focus on the most relevant parts, reducing wasted computational resources and improving the responsiveness to tasks.

[0069] Optionally, modal features of the target modal information input to each modal encoder are generated by modulating the self-attention mechanism of each modal encoder through the encoder control vector, including: modulating the query matrix of the self-attention mechanism through the spatial routing vector and a first projection bias term when the encoder control vector includes a spatial routing vector and a semantic routing vector; and modulating the key matrix of the self-attention mechanism through the semantic routing vector and a second projection bias term; wherein the spatial routing vector is used to indicate the position information that the user instruction requires attention in the target modal information, and the semantic routing vector is used to indicate the attribute information that the user instruction requires attention in the target modal information; and the modal features of the target modal information are determined through the attention mask matrix, the modulated query matrix, and the modulated key matrix.

[0070] Specifically, modulating the query matrix of the self-attention mechanism using the spatial routing vector and the first projection bias term includes: Modulating the key matrix of the self-attention mechanism using the semantic routing vector and the second projection bias term includes: ;in, Indicates the first projection bias term. This indicates the second projection bias term.

[0071] Then, the modal features of the target modal information are determined using the attention mask matrix, the modulated query matrix, and the modulated key matrix, including: Where Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; ConditionalAttention(Q,K,V|r) represents the modal features of the target modal information, and V represents the value matrix obtained after linear transformation of the target modal information. The dimension of the key vectors in the key matrix is ​​represented by , and M represents the attention mask matrix.

[0072] In this embodiment, the projection bias term allows prior knowledge to be incorporated into the network, guiding attention computation to favor certain query-key pairings, thereby strengthening or weakening specific modal interactions. The attention mask matrix further controls the attention mechanism, preventing the model from computation on specific positions or modalities. This is particularly suitable for processing multimodal data with temporal order or specific layouts, ensuring that the model follows the correct information flow and context when understanding and generating results, avoiding unnecessary interference, and improving decision quality and generalization ability.

[0073] Optionally, the fusion control vector controls the multimodal large model to fuse the modal features of the multimodal information corresponding to the user instruction, generating the interaction information corresponding to the user instruction. This includes: modal gating the modal features of the multimodal information using a modal weight vector to obtain gated features; and fusing the gated features using a fusion control vector to generate the interaction information corresponding to the user instruction. The fusion control vector includes the modal weight vector and the fusion control vector. The modal weight vector indicates the relative importance of each modal information to the user instruction, and the fusion control vector indicates the allowed fusion strategies for the multimodal information.

[0074] Each modal information is a modal information within the multimodal information.

[0075] Furthermore, modal gating is performed on the modal features of the multimodal information using modal weight vectors to obtain gated features, including: determining the gated features using the following formula: ;in, Indicates the features after gating. Modal features representing each modal information For the sigmoid function, , Let be the modal weight vector.

[0076] By modal gating the modal features of multimodal information, the contribution of different modal information can be dynamically adjusted, ensuring that the model focuses more on key modalities in specific tasks, thereby improving the accuracy and efficiency of multimodal task processing.

[0077] Furthermore, the interaction information corresponding to the user command is generated by fusing the gated features through the fusion control vector, including: determining the fusion strategy weight vector corresponding to the fusion control vector through a dynamic fusion processor; fusing the gated features through the fusion strategy weight vector to obtain a fusion vector; and generating the interaction information based on the fusion vector through the multimodal large model.

[0078] The process of generating the interaction information based on the fusion vector using the multimodal large model includes: inputting the fusion vector into the understanding and production core of the multimodal large model (which may be an LLM) to generate the interaction information.

[0079] Specifically, determining the fusion strategy weight vector corresponding to the fusion control vector through a dynamic fusion processor includes: determining the fusion strategy weight vector using the following formula, including: ;in, This represents the weight vector of the fusion strategy. This represents the fusion control vector, which is used to specify the fusion strategy.

[0080] Specifically, the gated features are fused using the fusion strategy weight vector to obtain a fusion vector, including: determining the fusion vector using the following formula: ;in, Represents the fusion vector. For the j-th predefined fusion strategy, The number of modes in multimodal information; This indicates the number of predefined fusion strategies. This represents the weight of the j-th fusion strategy. and .

[0081] The integration strategy includes: strong visual guidance, question-and-answer interaction, etc.

[0082] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the above-described interaction method based on a multimodal large model, the following description, in conjunction with embodiments, illustrates the process, but is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0083] This application provides a multimodal large model recognition method based on task-aware dynamic routing (equivalent to the multimodal large model-based interaction method in the above embodiments) and a system (used to implement the above-mentioned multimodal large model-based interaction method). By introducing a lightweight task-aware dynamic routing network, routing vectors are generated based on text instructions (i.e., user instructions), dynamically guiding the attention area of ​​the visual encoder, the interaction mode of the modality fusion unit, and the inference focus of the large language model, thus solving the problems of computational redundancy, modal interference, and intent understanding bias in existing multimodal large models. The method provided in this application proposes a "task-aware" dynamic multimodal information routing and fusion mechanism, realizing a paradigm shift from "static fusion" to "dynamic routing." This allows the multimodal large model to not process all tasks using a fixed pattern, but to dynamically generate a "customized" information processing pipeline based on the unique intent of each task. This is a fundamental paradigm shift.

[0084] Specifically, taking visual encoders as an example, "task-aware" visual encoding directly feeds back the task intent to the lowest level of visual encoding, achieving "on-demand perception." This fundamentally reduces redundant computation, improves processing speed, and enhances robustness to noise. Fine-grained fusion control: Through routing vectors, unprecedentedly fine control over the fusion process is possible, including modality importance weighting and fusion strategy selection, effectively avoiding negative interference between modalities. Modularity and lightweight: The routing network itself is lightweight; it does not change the backbone parameters of the large model but acts as a highly efficient "controller." This allows the routing network to be easily integrated into large multimodal models in related technologies without retraining from scratch, possessing high practical value.

[0085] In summary, the core idea of ​​this application's embodiments is to introduce a lightweight "task-aware dynamic routing network" before the encoding and fusion stages of traditional multimodal large models. This routing network generates a set of dynamic control signals (equivalent to the routing vector set in the above embodiments) based on text instructions (task intent) to guide the attention area of ​​the visual encoder, the interaction mode of the modality fusion unit, and even the inference focus of the large language model itself.

[0086] The system in this application embodiment mainly includes the following three core modules. Figure 3 This is a schematic diagram of the system architecture according to an embodiment of this application, such as... Figure 3 As shown, it includes:

[0087] Module 1: Task Intent Parsing and Route Vector Generation Module.

[0088] Input: The user's text command (e.g., "What is the red object in the top left corner of the image?").

[0089] Processing: Deep semantic parsing of user commands is performed using a shallow layer of a large language model or a dedicated lightweight text encoder.

[0090] Output: Generates a set of structured routing vectors (equivalent to the routing vector group in the above embodiment). This set of routing vectors includes:

[0091] Spatial routing vector: the spatial location (e.g., "top left corner") that the encoding task is concerned with.

[0092] Semantic routing vectors: conceptual attributes (such as "red" or "object") that the encoding task is concerned with.

[0093] Modality weight vector: Encodes the relative importance of each modality in completing this task (e.g., the visual modality weight is 0.9 and the text modality weight is 0.1 for this task).

[0094] Fusion control vector: Specifies the interaction strategy (such as "strong visual guidance", "question-and-answer interaction", etc.) that should be used in subsequent fusion layers.

[0095] Module 2: Dynamic Perception Encoding Module. Taking a visual encoder as an example:

[0096] Traditional approach: Visual encoders (such as ViT) perform undifferentiated feature extraction on the entire image (visual modal information in multimodal information).

[0097] Traditional self-attention formula: .

[0098] In the embodiments of this application: Figure 4 This is another flowchart of an optional interaction method based on a multimodal large model according to an embodiment of this application, such as... Figure 4 As shown, step S41: Input the image patch, spatial routing vector, and semantic routing vector of the original image into the actual encoder. Step S42: Conditional self-attention calculation. The spatial routing vector and semantic routing vector are injected into the attention calculation of the visual encoder, modifying the self-attention mechanism of the Vision Transformer to become a "conditional self-attention". The query and key are modulated through the routing vector, so that the visual encoder only invests high computational weight in image patches related to the task. Step S43: Generate a task-aware visual feature (equivalent to the modal feature of the target modality information in the above embodiment). For other modalities such as audio, a similar principle is used for dynamic modulation.

[0099] Specifically, the conditional self-attention introduced in this embodiment of the application after routing vector modulation includes:

[0100] ; ;

[0101] .

[0102] Optionally, the conditional self-attention after introducing route vector modulation also includes:

[0103] ; ;

[0104] .

[0105] The meanings of the symbols in the above-mentioned conditions related to attention are shown in Table 2.

[0106] Table 2

[0107]

[0108] Module 3: Dynamic Adaptive Fusion and Inference Module.

[0109] Traditional approach: Simply concatenate the processed multimodal features or fuse them using a fixed cross-attention module.

[0110] In the embodiments of this application: Figure 5 This is another flowchart of an optional interaction method based on a multimodal large model according to an embodiment of this application, such as... Figure 5 As shown, step S51: Multimodal features (equivalent to the modal features of multimodal information in the above embodiments), modal weight vectors, and fusion control vectors are input as control signals into a reconfigurable fusion network. This fusion network can dynamically adjust its internal connections based on these signals. Step S52: Modal gating. Based on the modal weight vectors, the features of secondary modalities are soft-masked or downweighted. Step S53: The reconfigurable fusion network (also called a dynamic fusion network) generates fused features (equivalent to the fusion vector in the above embodiments). The fusion control vector is used to select or interpolate from a variety of predefined micro-fusion strategies (equivalent to the fusion strategies in the above embodiments, such as visual question answering strategies, image description strategies, and fine-grained recognition strategies) to generate fusion network parameters most suitable for the current task. Finally, only the filtered and modulated information (equivalent to the fusion vector in the above embodiments) is fed into the large language model for final inference and text (equivalent to the interactive information in the above embodiments) generation.

[0111] Specifically, modal gating mechanisms include: Furthermore, modal gating mechanisms also include: The symbols in the formulas related to modal gating mechanisms have the following meanings: Table 3.

[0112] Table 3

[0113]

[0114] Dynamic converged networks include: ; ;in, For fusion weights. Dynamic fusion networks also include: ; The symbols in the formulas related to dynamic fusion networks are shown in Table 4.

[0115] Table 4

[0116]

[0117] Optionally, the model construction of the multimodal large model in this embodiment includes: using an open-source visual language large model (such as LLaVA (Large Language and Vision Assistant)) as the base model. Routing network: using a lightweight Transformer encoder to receive text instructions and output four routing vectors. This can be achieved by fine-tuning the text encoder of the base large model using LORA (Low-Rank Adaptation). Dynamic visual encoding: in the basic ViT, the spatial routing vector and semantic routing vector are multiplied (modulated) element-wise with the Query and Key vectors respectively, and then the attention score is calculated. Dynamic fusion processor: a small MLP is designed, taking the fusion control vector as input and outputting a set of weights (equivalent to the fusion strategy weight vector in the above embodiment). These weights are used to weighted summation of the outputs of several basic fusion kernels (such as linear layers and cross-attention layers), thereby achieving dynamic reconstruction of the fusion strategy.

[0118] Furthermore, the training process for large multimodal models includes:

[0119] Phase 1 (Freeze Pre-training): Freeze all parameters of the base large model (LLaVA) and train only the reconfigurable parts of the routing network and dynamic fusion unit. Train using multimodal instruction tracking data (such as LLaVA-Instruct) and the standard text generation loss function.

[0120] The second phase (joint fine-tuning) involves unfreezing parts of the base model (e.g., freezing only the underlying visual encoder) and performing end-to-end joint fine-tuning with the routing network to further optimize performance.

[0121] Optionally, one application example of this application embodiment includes:

[0122] The user input command is: "Compare the differences in sky color between these two images." The routing network parses the core task as "comparison," "sky," and "color." The generated routing vectors guide the visual encoders of both images to focus on the "sky" region; during the fusion phase, they assign extremely high weights to the visual modalities of the two images and activate the "comparative analysis" fusion strategy. The final output is a precise comparative description generated by the multimodal large model, rather than independent descriptions of the two images. It is evident that the core competitiveness of each embodiment lies in its "dynamic nature" and "task awareness," directly addressing the pain points of the current multimodal large model's "static" and "one-size-fits-all" architecture. This endows the model with the ability to "think before acting," first understanding the task and then dynamically allocating resources to complete it.

[0123] Optionally, another application example of this application embodiment includes:

[0124] The task is to determine "What is the red object in the upper left corner of the image?"

[0125] Step 1: Route vector generation.

[0126] Enter text (user command): "What is the red object in the upper left corner of the image?"

[0127] Routing network output:

[0128] Spatial routing vector A 196-dimensional vector, with a weight of 0.8 in the top-left corner and uniformly distributed weights in other areas. Semantic routing vector. A 1024-dimensional vector encoding semantic concepts such as "red" and "object". Modal weight vector. [0.9, 0.1], visual modality weight 0.9, text modality weight 0.1. Fusion control vector. : 256-dimensional vector, biased towards question-and-answer interaction strategy.

[0129] Step 2: Calculate conditional self-attention.

[0130] Assume the parameters of the ViT encoder include: sequence length n = 197 (196 patches + 1 CLS token (ClassToken)); feature depth d = 760; number of attention heads h = 12; =768 / 12=64.

[0131] The specific calculation process includes:

[0132] Projecting a 196-dimensional spatial vector onto a 768-dimensional vector.

[0133] Projecting a 1024-dimensional semantic vector onto a 768-dimensional vector.

[0134] Modulate the query to enhance attention in the top left corner area.

[0135] Modulate the key to enhance features associated with the "red object".

[0136] Step 3: Modal gating calculation.

[0137] Visual features : 768-dimensional vector.

[0138] Text features : 768-dimensional vector.

[0139] Gating calculation:

[0140] .

[0141] Using hard sigmoid: .

[0142] Step 4: Selecting a dynamic fusion strategy.

[0143] Fusion control vector processing: The fusion weights are obtained after Softmax: .

[0144] Final Fusion: ;

[0145] in This is a cross-attention strategy, best suited for question-answering tasks.

[0146] The embodiments of this application significantly improve computational efficiency: through task-aware visual encoding, computation in many task-irrelevant areas can be skipped, resulting in a 30%-50% increase in inference speed when handling specific question-and-answer and referential tasks. It also improves recognition and inference accuracy: by suppressing modal interference and noise and precisely guiding the model's focus, accuracy is significantly improved in tasks requiring fine-grained understanding (such as VQA and referential localization). Furthermore, it enhances model transparency and interpretability: the generation process of routing vectors can be visualized (e.g., showing which parts of the image the model focuses on), making the model's decision-making process more transparent. Finally, it provides better generalization ability: because the model (multimodal large model for short) learns to dynamically adjust its strategy according to the task, it exhibits stronger adaptability and zero-shot learning ability for complex combinations of unseen tasks.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0148] This embodiment also provides an interactive device based on a multimodal large model, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0149] Figure 6 This is a structural block diagram of an optional interactive device based on a multimodal large model according to an embodiment of this application. The device includes:

[0150] The parsing module 62 is used to perform deep semantic parsing of user commands through a routing network to generate a routing vector group; wherein, the routing vector group includes: encoder control vector and fusion control vector, and the encoder control vector is used to control at least one modal encoder in the multimodal large model;

[0151] The modulation module 64 is used to modulate the self-attention mechanism of each of the at least one modal encoders through the encoder control vector to generate modal features of the target modal information input to each modal encoder;

[0152] The control module 66 is used to control the multimodal large model to fuse the modal features of the multimodal information corresponding to the user command through the fusion control vector, and generate the interaction information corresponding to the user command; wherein, the modal features of the multimodal information include: the modal features of the target modal information.

[0153] The aforementioned device performs deep semantic parsing of user commands through a routing network to generate a routing vector set. This routing vector set includes an encoder control vector and a fusion control vector. The encoder control vector controls at least one modal encoder in the multimodal large model. The encoder control vector modulates the self-attention mechanism of each modal encoder, generating modal features of the target modal information input to each modal encoder. The fusion control vector controls the multimodal large model to fuse the modal features of the multimodal information corresponding to the user command, generating interactive information corresponding to the user command. The modal features of the multimodal information include the modal features of the target modal information. This ensures that the generation and fusion of modal features in the multimodal large model's processing of multimodal information are controlled by the routing vector set obtained from deep semantic parsing of the user command. Therefore, this technical solution solves the problem in related technologies where multimodal large models cannot dynamically process multimodal information based on user commands. It achieves the technical effect of strongly correlating the processing of multimodal information by the multimodal large model with user commands.

[0154] In an exemplary embodiment, the parsing module 62 is further configured to convert the user instruction into a target vector using a text encoder and a multilayer perceptron; slice the target vector based on a preset dimension using a vector decomposition layer to obtain multiple slice vectors; process each slice vector according to the activation function corresponding to each slice vector to obtain the routing vector group; wherein, the routing network includes: the text encoder, a multilayer perceptron connected to the text encoder, and a vector decomposition layer connected to the multilayer perceptron.

[0155] In an exemplary embodiment, the modulation module 64 is further configured to modulate the query matrix and key matrix of the self-attention mechanism respectively through the encoder control vector; and determine the modal features of the target modal information through the modulated query matrix and the modulated key matrix.

[0156] In an exemplary embodiment, the modulation module 64 is further configured to determine the modulated query matrix and the modulated key matrix respectively through the spatial routing vector and the semantic routing vector in the encoder control vector; wherein, the spatial routing vector is used to indicate the position information that the user instruction requires to be noticed in the target modal information, and the semantic routing vector is used to indicate the attribute information that the user instruction requires to be noticed in the target modal information.

[0157] In an exemplary embodiment, the modulation module 64 is further configured to determine the modulated query matrix using the following formula, when the encoder control vector includes a spatial routing vector and a semantic routing vector: The modulated bond matrix is ​​determined using the following formula: Wherein, Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; and For learnable projection matrices, This indicates element-wise multiplication; For spatial routing vectors, This is a semantic routing vector.

[0158] In an exemplary embodiment, the modulation module 64 is further configured to determine the modal characteristics of the target modal information using the following formula, including: Where Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; ConditionalAttention(Q,K,V|r) represents the modal features of the target modal information, and V represents the value matrix obtained after linear transformation of the target modal information. This represents the dimension of the key vectors in the key matrix.

[0159] In an exemplary embodiment, the modulation module 64 is further configured to, when the encoder control vector includes a spatial routing vector and a semantic routing vector, modulate the query matrix of the self-attention mechanism using the spatial routing vector and a first projection bias term; and modulate the key matrix of the self-attention mechanism using the semantic routing vector and a second projection bias term; wherein the spatial routing vector is used to indicate the location information that the user instruction requires attention in the target modal information, and the semantic routing vector is used to indicate the attribute information that the user instruction requires attention in the target modal information; and the modal features of the target modal information are determined using the attention mask matrix, the modulated query matrix, and the modulated key matrix.

[0160] In one exemplary embodiment, the modulation module 64 is further configured to Where Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; ConditionalAttention(Q,K,V|r) represents the modal features of the target modal information, and V represents the value matrix obtained after linear transformation of the target modal information. The dimension of the key vectors in the key matrix is ​​represented by , and M represents the attention mask matrix.

[0161] In an exemplary embodiment, the control module 66 is further configured to perform modal gating on the modal features of the multimodal information using a modal weight vector to obtain gated features; and to fuse the gated features using a fusion control vector to generate interactive information corresponding to the user instruction; wherein the fusion control vector includes: the modal weight vector and the fusion control vector; the modal weight vector is used to indicate the relative importance of each modal information to the user instruction, and the fusion control vector is used to indicate the fusion strategy allowed to be adopted for the multimodal information.

[0162] In one exemplary embodiment, the control module 66 is further configured to determine the gated features using the following formula: ;in, Indicates the features after gating. Modal features representing each modal information For the sigmoid function, , Let be the modal weight vector.

[0163] In an exemplary embodiment, the control module 66 is further configured to determine the fusion strategy weight vector corresponding to the fusion control vector through a dynamic fusion processor; fuse the gated features through the fusion strategy weight vector to obtain a fusion vector; and generate the interaction information based on the fusion vector through the multimodal large model.

[0164] In an exemplary embodiment, the control module 66 is further configured to determine the fusion strategy weight vector using the following formula: ;in, This represents the weight vector of the fusion strategy. This represents the fusion control vector, which is used to specify the fusion strategy.

[0165] In an exemplary embodiment, the control module 66 is further configured to determine the fusion vector using the following formula, including: ;in, Represents the fusion vector. For the j-th predefined fusion strategy, The number of modes in multimodal information; This indicates the number of predefined fusion strategies. This represents the weight of the j-th fusion strategy. and .

[0166] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0167] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0168] S1, deep semantic parsing of user commands is performed through a routing network to generate a routing vector group; wherein, the routing vector group includes: encoder control vector and fusion control vector, the encoder control vector being used to control at least one modal encoder in the multimodal large model;

[0169] S2, modal features of the target modal information input to each modal encoder are generated by modulating the self-attention mechanism of each modal encoder in the at least one modal encoder through the encoder control vector.

[0170] S3, the multimodal large model is controlled by the fusion control vector to fuse the modal features of the multimodal information corresponding to the user command, and generate the interaction information corresponding to the user command; wherein, the modal features of the multimodal information include: the modal features of the target modal information.

[0171] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0172] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0173] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0174] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0175] S1, deep semantic parsing of user commands is performed through a routing network to generate a routing vector group; wherein, the routing vector group includes: encoder control vector and fusion control vector, the encoder control vector being used to control at least one modal encoder in the multimodal large model;

[0176] S2, modal features of the target modal information input to each modal encoder are generated by modulating the self-attention mechanism of each modal encoder in the at least one modal encoder through the encoder control vector.

[0177] S3, the multimodal large model is controlled by the fusion control vector to fuse the modal features of the multimodal information corresponding to the user command, and generate the interaction information corresponding to the user command; wherein, the modal features of the multimodal information include: the modal features of the target modal information.

[0178] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0179] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0180] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0181] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0182] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0183] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0184] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An interaction method based on a multimodal large model, characterized in that, include: The routing network performs deep semantic parsing of user commands to generate a routing vector set; wherein the routing vector set includes: encoder control vector and fusion control vector, the encoder control vector is used to control at least one modal encoder in the multimodal large model, and the routing network includes: text encoder, multilayer perceptron connected to the text encoder, and vector decomposition layer connected to the multilayer perceptron; The self-attention mechanism of each modal encoder in the at least one modal encoder is modulated by the encoder control vector to generate modal features of the target modal information input to each modal encoder; The fusion control vector controls the multimodal large model to fuse the modal features of the multimodal information corresponding to the user command, thereby generating the interaction information corresponding to the user command; wherein, the modal features of the multimodal information include: the modal features of the target modal information.

2. The interaction method based on a multimodal large model according to claim 1, characterized in that, The user commands are subjected to deep semantic parsing through the routing network to generate a set of routing vectors, including: The user instructions are converted into target vectors using a text encoder and a multilayer perceptron. The target vector is sliced ​​based on a preset dimension by a vector decomposition layer to obtain multiple slice vectors; Each slice vector is processed according to the activation function corresponding to each slice vector to obtain the routing vector group.

3. The interaction method based on a multimodal large model according to claim 1, characterized in that, Modal features of the target modal information input to each modal encoder are generated by modulating the self-attention mechanism of each modal encoder in the at least one modal encoder through the encoder control vector modulation, including: The query matrix and key matrix of the self-attention mechanism are modulated by the encoder control vector, respectively. The modal features of the target modal information are determined by the modulated query matrix and the modulated key matrix.

4. The interaction method based on a multimodal large model according to claim 3, characterized in that, The query matrix and key matrix of the self-attention mechanism are modulated by the encoder control vector, including: The modulated query matrix and the modulated key matrix are determined by the spatial routing vector and the semantic routing vector in the encoder control vector, respectively. The spatial routing vector is used to indicate the location information that the user instruction requires attention in the target modality information, and the semantic routing vector is used to indicate the attribute information that the user instruction requires attention in the target modality information.

5. The interaction method based on a multimodal large model according to claim 3, characterized in that, The query matrix and key matrix of the self-attention mechanism are modulated by the encoder control vector, including: When the encoder control vector includes a spatial routing vector and a semantic routing vector, the modulated query matrix is ​​determined by the following formula: The modulated bond matrix is ​​determined using the following formula: ; Wherein, Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; and For learnable projection matrices, This indicates element-wise multiplication; For spatial routing vectors, This is a semantic routing vector.

6. The interaction method based on a multimodal large model according to claim 3, characterized in that, The modal features of the target modal information are determined by the modulated query matrix and the modulated key matrix, including: The modal features of the target modal information are determined by the following formulas, including: ; Where Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; ConditionalAttention(Q,K,V|r) represents the modal features of the target modal information, and V represents the value matrix obtained after linear transformation of the target modal information. This represents the dimension of the key vectors in the key matrix.

7. The interaction method based on a multimodal large model according to claim 1, characterized in that, Modal features of the target modal information input to each modal encoder are generated by modulating the self-attention mechanism of each modal encoder in the at least one modal encoder through the encoder control vector modulation, including: When the encoder control vector includes a spatial routing vector and a semantic routing vector, the query matrix of the self-attention mechanism is modulated by the spatial routing vector and a first projection bias term; and the key matrix of the self-attention mechanism is modulated by the semantic routing vector and a second projection bias term; wherein, the spatial routing vector is used to indicate the positional information that the user instruction requires attention in the target modal information, and the semantic routing vector is used to indicate the attribute information that the user instruction requires attention in the target modal information; The modal features of the target modal information are determined by the attention mask matrix, the modulated query matrix, and the modulated key matrix.

8. The interaction method based on a multimodal large model according to claim 7, characterized in that, The modal features of the target modal information are determined by the attention mask matrix, the modulated query matrix, and the modulated key matrix, including: ； Where Q represents the query matrix, Q' represents the modulated query matrix, K represents the key matrix, and K' represents the modulated key matrix; ConditionalAttention(Q,K,V|r) represents the modal features of the target modal information, and V represents the value matrix obtained after linear transformation of the target modal information. Let M represent the dimension of the key vectors in the key matrix, and M represent the attention mask matrix.

9. The interaction method based on a multimodal large model according to claim 1, characterized in that, The fusion unit controls the multimodal large model to fuse the modal features of the multimodal information corresponding to the user command through the fusion unit control vector, generating the interaction information corresponding to the user command, including: Modal gating is performed on the modal features of the multimodal information using modal weight vectors to obtain gated features; By fusing the gated features using the fusion control vector, interactive information corresponding to the user command is generated. The fusion control vector includes the modality weight vector and the fusion control vector; the modality weight vector is used to indicate the relative importance of each modality information to the user instruction, and the fusion control vector is used to indicate the fusion strategy allowed for the multimodal information.

10. The interaction method based on a multimodal large model according to claim 9, characterized in that, Modal gating is performed on the modal features of the multimodal information using modal weight vectors to obtain gated features, including: The gating features are determined using the following formula: ; in, Indicates the features after gating. Modal features representing each modal information For the sigmoid function, , Let be the modal weight vector.

11. The interaction method based on a multimodal large model according to claim 9, characterized in that, By fusing the gated features using the fusion control vector, interactive information corresponding to the user command is generated, including: The fusion strategy weight vector corresponding to the fusion control vector is determined by a dynamic fusion processor. The gated features are fused using the weight vector of the fusion strategy to obtain a fusion vector; The interaction information is generated based on the fusion vector using the multimodal large model.

12. The interaction method based on a multimodal large model according to claim 11, characterized in that, The fusion strategy weight vector corresponding to the fusion control vector is determined by a dynamic fusion processor, including: The fusion strategy weight vector is determined by the following formula, including: ； in, This represents the weight vector of the fusion strategy. This represents the fusion control vector, which is used to specify the fusion strategy.

13. The interaction method based on a multimodal large model according to claim 11, characterized in that, The gated features are fused using the weight vector of the fusion strategy to obtain a fusion vector, including: The fusion vector is determined by the following formula, including: ； in, Represents the fusion vector. For the j-th predefined fusion strategy, The number of modes in multimodal information; This indicates the number of predefined fusion strategies. This represents the weight of the j-th fusion strategy. and .

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 13.

15. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 13 through the computer program.

Images