Text generation method and related device

Abstract

The present disclosure provides a text generation method and related equipment, the method comprising: obtaining an image and text data, the text data including attribute constraint conditions of the text; extracting initial features of the image and the text data, mapping the initial features of the image and the initial features of the text to a shared feature space to obtain multi-modal fusion features; inputting the multi-modal fusion features into an encoder of a pre-trained large language model, the encoder including multiple hierarchical encoding layers, the multiple hierarchical encoding layers including at least two hierarchical groups, calculating routing weights of each hierarchical group based on a routing network corresponding to each hierarchical group, and weighting and fusing hierarchical features output by each hierarchical group according to the routing weights to obtain hierarchical fusion features; inputting the hierarchical fusion features into a low-rank expert subnetwork group of the large language model to obtain target encoding features; constructing an embedding vector corresponding to the attribute constraint conditions, and inputting the target encoding features fused with the embedding vector into a decoder of the large language model to obtain a target text.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a text generation method and related equipment. Background Technology

[0002] LLaVA (Large Language and Vision Assistant) is a mainstream multimodal large language model. Its architecture aligns the image features output by the visual encoder to the text semantic space through a learnable projection layer, and then concatenates them with text instructions before inputting them into the language model to achieve unified visual-language understanding and generation. The model employs a two-stage training strategy: the first stage uses large-scale image-text pairs to train cross-modal alignment capabilities, and the second stage fine-tunes the instructions based on high-quality visual instruction data, enabling it to perform complex multimodal tasks such as visual question answering and reasoning description.

[0003] However, the current LLaVA model architecture lacks an explicit attribute control module. Its text generation process is entirely dominated by the content of the input image, resulting in LLaVA models only being able to generate neutral, objective descriptive text. When user input includes fine-grained text generation attribute constraints (such as requiring text to be output from a specific perspective or in a specific style), the current LLaVA model cannot recognize and respond to these constraints. The generated results fail to meet users' personalized needs for text perspective, style, and other personalized attributes, significantly limiting its applicability. Summary of the Invention

[0004] This disclosure proposes a text generation method and related equipment to solve or partially solve the above-mentioned problems.

[0005] This disclosure provides a text generation method, comprising: acquiring an image and text data, wherein the text data includes attribute constraints of the text; extracting initial features from the image and the text data respectively to obtain initial image features and initial text features; mapping the initial image features and the initial text features to a shared feature space of the same dimension to obtain aligned multimodal fusion features; and inputting the multimodal fusion features into an encoder of a pre-trained large language model, wherein the encoder includes multiple levels of encoding layers, each level including at least two level groups, based on a preset set of parameters corresponding to each level group. The routing network calculates the routing weights for each of the aforementioned hierarchical groups, and performs weighted fusion of the hierarchical features output by each of the aforementioned hierarchical groups based on the routing weights to obtain hierarchical fused features. The routing weights are used to characterize the importance of the features of the corresponding hierarchical group to the current encoding task. The hierarchical fused features are input into the low-rank expert subnetwork group of the large language model to obtain target encoding features. The low-rank expert subnetwork group includes at least two low-rank expert subnetworks. An embedding vector corresponding to the attribute constraints is constructed, and the target encoding features are fused with the embedding vector and input into the decoder of the large language model to obtain the target text.

[0006] A second aspect of this disclosure provides a computer device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the one or more programs include instructions for performing the method of the first aspect.

[0007] A third aspect of this disclosure provides a non-volatile computer-readable storage medium comprising a computer program that, when executed by one or more processors, causes the one or more processors to perform the method described in the first aspect.

[0008] The fourth aspect of this disclosure provides a computer program product comprising one or more computer programs that, when executed by one or more processors, implement the method as described in the first aspect.

[0009] The text generation method of this disclosure obtains multimodal fusion features by extracting, mapping, and aligning features from image and text data, thereby converting heterogeneous image and text data into a unified feature representation that can be understood by a large language model. The multimodal fusion features are input into the encoder of a pre-trained large language model, and the routing weights of each level group are calculated based on the routing network of each level group of the encoder. The layer-level features output by each level group are then weighted and fused according to the calculated routing weights to obtain layer-level fusion features. These layer-level fusion features are then input into a low-rank expert sub-network group for fine-tuning to obtain target encoding features. Finally, an embedding vector constructed based on attribute constraints is used as a query vector to interact with the target encoding features, dynamically fusing the deep semantic information of the image and the text to generate the target text. Since the encoder of the large language model calculates routing weights for each level group based on a preset routing network, rather than using a uniform weight, interference information unrelated to the target attribute in the attribute constraints can be eliminated by defining the routing strategy function in the preset routing network. This enables the controllability of the output text attributes, allowing the model to generate text that is highly relevant to the text and visual context and has specific attributes based on image and text data. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic diagram of an exemplary system provided by an embodiment of this disclosure is shown.

[0012] Figure 2 A flowchart illustrating an exemplary text generation method provided in an embodiment of this disclosure is shown.

[0013] Figure 3 A schematic diagram of the structure of an exemplary encoder provided in an embodiment of this disclosure is shown.

[0014] Figure 4 A schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0016] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0017] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0018] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0019] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0020] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0021] Figure 1 A schematic diagram of an exemplary system 100 provided in an embodiment of this disclosure is shown.

[0022] like Figure 1As shown, system 100 may include terminal device 102, terminal device 104, server 106, and database server 108. A medium (e.g., a network) may be included between terminal device 102, terminal device 104, server 106, and database server 108 to provide a communication link. This network may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0023] Various software or applications (APPs) can be installed on terminal devices 102 and 104, such as image processing software or applications, video conferencing software or applications, reading software or applications, video software or applications, social networking software or applications, payment software or applications, web browsers, and instant messaging tools. In some embodiments, these software or applications can all be used for text generation.

[0024] The terminal devices 102 and 104 here can be hardware or software. When terminal devices 102 and 104 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 players, laptops, and desktop computers (PCs). When terminal devices 102 and 104 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are made here.

[0025] Server 106 can be a server that provides various services, such as a backend server that supports various applications displayed on terminal devices 102 and 104. Database server 108 can also be a database server that provides various services. It is understood that if server 106 can implement the relevant functions of database server 108, database server 108 may not need to be set up in system 100.

[0026] The server 106 and database server 108 here can be either hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0027] It should be noted that the text generation method provided in this embodiment can be executed by terminal device 102, terminal device 104, or server 106, or interactively executed by the various devices in system 100. It should be understood that... Figure 1The number of terminal devices, users, servers, and database servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, users, servers, and database servers.

[0028] As an example scenario, when user 110 inputs instructions containing images and text to terminal device 102, and user 112 inputs instructions containing images and text to terminal device 104, terminal device 102 generates target text based on the LLaVA model deployed on it, according to the images and text input by user 110, and terminal device 104 generates target text based on the LLaVA model deployed on it, according to the images and text input by user 112. The process of generating target text by the LLaVA model is entirely dominated by the content of the input images, resulting in it being able to generate only neutral and objective descriptive text. When the instructions input by user 110 or user 112 contain fine-grained text generation attribute constraints (such as requiring the text to be output with a certain stance, style, or theme), the LLaVA model cannot recognize and respond to the text generation attribute constraints, and the generated results are difficult to meet the user's personalized needs for text stance, style, or theme, etc., resulting in significant limitations in applicable scenarios.

[0029] In view of this, embodiments of the present disclosure provide a text generation method to solve or partially solve the above-mentioned problems.

[0030] Figure 2 A flowchart illustrating an exemplary text generation method 200 provided in this disclosure is shown. This method 200 can be applied to text generation in scenarios such as social media sentiment guidance, digital marketing copywriting, or intelligent comment assistants. This method can understand multimodal input containing text and images (e.g., a tweet with both text and images) and automatically generate high-quality, firmly committed, and context-appropriate replies or comments based on the user-specified stance (support / oppose) or language style (e.g., humor, sarcasm). Optionally, this method 200 can be... Figure 1 Terminal devices 102 and 104 can be implemented separately, or they can be... Figure 1 The server 106 can be used to implement this, or it can be implemented by... Figure 1 The interaction between devices in System 100 is implemented. For example... Figure 2 As shown, the method includes the following processing: In step 202, image and text data are acquired, wherein the text data includes text attribute constraints. Optionally, the attribute constraints of the text may include the target attributes of the generated text, which include, but are not limited to, the stance of the generated text (such as for or against), language style (humor or satire), and topic.

[0031] In step 204, initial features of the image and the text data are extracted respectively to obtain initial image features and initial text features; Optionally, a multimodal dataset containing images, text, and target attribute labels can be constructed first. The data can be loaded in JSON (JavaScript Object Notation) format. Taking an application scenario that generates target text based on forum post content as an example, each data entry can contain post text, image paths, target stance (stance, such as support / opposition), and style instructions (style, such as irony / humor). For each input instance... (in, For post text, For image, (For positional instructions), construct an instruction sequence that conforms to the input specifications of a large language model. The construction logic of the input sequence is as follows: ; in, Indicates sequence concatenation. It is a placeholder token (the smallest semantic unit in natural language processing) used to fill in visual features later. and Used for labeling visual regions. A pre-trained tokenizer can convert the constructed text sequence into a token ID (lexical identifier) ​​sequence and generate the corresponding attention mask.

[0032] The generated Token ID sequence is input into the text embedding layer of the large language model and processed based on the following expression to obtain the text embedding sequence (an example of the initial text features mentioned above): ; In the above formula, It is the sequence length. This is the hidden layer dimension of the model (e.g., 4096). This process can convert discrete text symbols into continuous numerical vector representations. The word segmenter processes the input text sequence arrive Process it.

[0033] In step 206, the initial image features and the initial text features are mapped to a shared feature space of the same dimension to obtain aligned multimodal fusion features; To enable large language models to understand image information, pre-trained visual encoders, such as CLIP-ViT (Contrastive Language-Image Pretraining-Vision Transformer), can be used as visual towers to interpret the input image based on the following expression. Perform feature extraction: ; in, It is the original visual feature, dimension (For example, 1024) may correspond to the text embedding dimension. Inconsistency. To achieve feature alignment, VisionTower(V): refers to the VisionTower module processing the input visual data V (such as images or video frames), which maps visual features to the text feature space through a multimodal projector. This projector consists of two linear layers and a GeLU activation function, expressed as follows: ; In the above formula, and It is a learnable projection matrix. It is a bias term. After projection, the visual features The dimension was adjusted to This is consistent with text features.

[0034] Aligned visual features Injected into text embedding sequence The specific operation is to find the sequence... Mark the corresponding position index And replace or add visual features: Ultimately, a joint feature sequence integrating visual perception and textual semantics is formed. (This is an example of the multimodal fusion features described above). This sequence will serve as input to subsequent Transformer layers for dynamic route computation.

[0035] In step 208, the multimodal fusion features are input into the encoder of the pre-trained large language model. The encoder includes multi-level coding layers, and each multi-level coding layer includes at least two level groups. The routing weights of each level group are calculated based on a preset routing network corresponding to each level group. The level features output by each level group are weighted and fused according to the routing weights to obtain the level fusion features. The routing weights are used to characterize the importance of the features of the corresponding level group to the current coding task. In one or more embodiments of this disclosure, based on the characteristic that semantic features in a large language model gradually become more abstract as the network depth increases, the multiple coding layers contained in the model are logically divided into multiple consecutive processing stages. Each processing stage corresponds to a layer group, and a specific routing policy function is defined for the routing network of each layer group. This allows each processing stage to determine the routing weight based on the routing network corresponding to each layer group. In other words, each layer group can calculate the routing weight of that layer based on the corresponding routing network. Thus, the layer features output by each layer group can be weighted and fused according to the routing weight to obtain the layer fused features.

[0036] In step 210, the hierarchical fusion features are input into the low-rank expert subnetwork group of the large language model to obtain the target encoded features. The low-rank expert subnetwork group includes at least two low-rank expert subnetworks. Optionally, without updating the weights of the dense layers of the pre-trained large language model, a parallel low-rank adaptation (LoRA) branch can be constructed for each expert subnetwork in each layer to achieve efficient parameter fine-tuning. Based on this, the hierarchical fusion features are input into the low-rank expert subnetwork group, and the activation weights of each low-rank expert subnetwork are calculated through a gating network. Each low-rank expert subnetwork performs refined feature extraction on the hierarchical fusion features through low-rank matrix factorization to obtain expert features. All expert features are weighted and aggregated according to the activation weights to output the final encoded features.

[0037] In step 212, an embedding vector corresponding to the attribute constraint is constructed, and the target encoding feature is fused with the embedding vector and then input into the decoder of the large language model to obtain the target text.

[0038] Optionally, after fusing the target encoded features with the embedding vector and inputting them into the decoder of the large language model, the attribute constraints can be used to adjust the weighted original prediction score vector (i.e., Logits) at the token level output by the large language model. This will enhance the scores of tokens that match the target attributes in the attribute constraints and reduce the scores of tokens that conflict with the target attributes, thereby adjusting the Logits. After normalization, the adjusted Logits will enable the model to generate target text that satisfies the attribute constraints.

[0039] The text generation method of this disclosure obtains multimodal fusion features by extracting, mapping, and aligning features from image and text data, thereby converting heterogeneous image and text data into a unified feature representation that can be understood by a large language model. The multimodal fusion features are input into the encoder of a pre-trained large language model, and the routing weights of each level group are calculated based on the routing network of each level group of the encoder. The layer-level features output by each level group are then weighted and fused according to the calculated routing weights to obtain layer-level fusion features. These layer-level fusion features are then input into a low-rank expert sub-network group for fine-tuning to obtain target encoding features. Finally, an embedding vector constructed based on attribute constraints is used as a query vector to interact with the target encoding features, dynamically fusing the deep semantic information of the image and the text to generate the target text. Since the encoder of the large language model calculates routing weights for each level group based on a preset routing network, rather than using a uniform weight, interference information unrelated to the target attribute in the attribute constraints can be eliminated by defining the routing strategy function in the preset routing network. This enables the controllability of the output text attributes, allowing the model to generate text that is highly relevant to the text and visual context and has specific attributes based on image and text data.

[0040] In one or more embodiments of this disclosure, based on the characteristic that semantic features in a large language model gradually become more abstract as the network depth increases, the multiple coding layers contained in the model can be logically divided into three consecutive processing stages, each processing stage corresponding to a hierarchical group, and a specific routing strategy function is defined for each processing stage. Figure 3 A schematic diagram of the structure of an encoder 300 for an exemplary large language model provided in this disclosure embodiment is shown, as follows: Figure 3 As shown, the multi-level coding layers of encoder 300 may include: a first level group 32, a second level group 34, and a third level group 36. Based on this, a routing weight for each level group is calculated based on a preset routing network corresponding to each level group. The hierarchical features output by each level group are weighted and fused according to the routing weights to obtain hierarchical fusion features, which may specifically include: Lexical and syntactic features of the multimodal fusion features are extracted based on the expert subnetworks in the first-level group, wherein the difference between the weight of each expert subnetwork in the first-level group and the average weight does not exceed a first threshold. Weights are assigned to the expert subnetworks in the second-level group based on the attention mechanism. The lexical features and the syntactic features are then weighted and fused according to the weights of each expert subnetwork in the second-level group to obtain the fused features. Weights are assigned to the expert subnetworks in the third-level group based on the attention mechanism. Only the target expert subnetworks in the third-level group whose semantic relevance to the target attribute in the attribute constraint reaches the second threshold are activated. The fusion features are processed based on the target expert subnetworks to obtain and output the hierarchical fusion features.

[0041] like Figure 3 As shown, the first layer group is the bottom layer (layers 0-7) of the encoder, corresponding to the preceding layers of the model. It is used to extract basic lexical and syntactic features of the input sequence. A uniform mixing strategy is adopted, focusing on the extraction of general features, which can ensure the stability and generalization of feature extraction and prevent premature overfitting. The uniform mixing strategy means that no expert sub-network selection is performed, and all expert sub-networks in this part participate in the calculation with approximately equal weights. The second layer group is the middle layer (layers 8-15) of the encoder, corresponding to the middle layers of the model. It is used for deep fusion of multimodal semantics and contextual interaction. A weighted mixing strategy is adopted, focusing on the fusion of multimodal semantics. This stage retains all expert sub-networks in this part and assigns different weights to each expert sub-network according to the attention mechanism, which can dynamically integrate visual and textual information according to the input context. The third level group is the high-level part of the encoder (layers 16-31), corresponding to the subsequent layers of the model. It is used to generate high-level target attributes and adopts a sparse hybrid strategy. It focuses on reasoning about specific attributes and only activates a very small number of expert subnetworks that are highly related to the target attributes. This can eliminate interference information that is unrelated to the target attributes and strengthen the control over the target attributes.

[0042] The embodiments of this disclosure employ a dynamic routing mechanism to dynamically adjust the hybridization strategy of the expert sub-network models based on the semantic depth of different levels of the encoder. At the bottom layer of the model, a uniform hybridization strategy is used to ensure that all expert sub-networks participate in the computation, thereby guaranteeing the fluency of the generated text and cross-modal alignment. In the middle layer of the model, an attention-guided weighted routing mechanism is introduced, using the embedding vector of the target attribute as a query signal to interact with the features of the expert sub-networks, thus dynamically fusing the deep semantic information of the image and text. At the high layer of the model, a sparse hybridization strategy is used to activate only a very small number of expert sub-networks highly correlated with the target attribute, thereby eliminating irrelevant interference information and achieving precise control over the attributes of the generated text. This hierarchical combination approach, from dense to sparse, ensures that the model can strictly follow preset attribute instructions to generate text responses with specific attributes while understanding complex multimodal contexts.

[0043] Optionally, weights may be assigned to the expert subnetworks in the second or third level group based on an attention mechanism, specifically including: Project the input hidden state of the first coding layer containing the expert subnetwork into a routing query vector; Set the key vector for the first coding layer; Based on the routing query vector and the key vector, the routing score of the expert subnetwork is calculated using the dot product attention mechanism. The routing score is then normalized to obtain the basic weights of the expert subnetwork in the first coding layer.

[0044] The following example illustrates the process of assigning weights to expert subnetworks in the second or third level group, using the formula.

[0045] Assume the first The input hidden state of the layer is First, the hidden state is projected into a routing query vector through a linear transformation. At the same time, for the first Each expert subnetwork of the layer Pre-set or learn a key vector Calculate the first... using the dot product attention mechanism The routing scores of each expert subnetwork are calculated, and the basic weights are obtained by normalizing them using the Softmax function. The expression is as follows: ; In the above formula, Let be the dimension of the key vector. This formula ensures that the sum of the weights is 1, and that the weight values ​​dynamically reflect the semantic relevance of the expert subnetwork to the current multimodal context (the query vector containing instructions with attribute constraints).

[0046] Optionally, only the target expert subnetwork in the third-level group whose semantic relevance to the target attribute in the attribute constraints reaches the second threshold may be activated, specifically including: Select the expert subnetworks whose basic weights are greater than the preset sparsity threshold as the target expert subnetworks to obtain the set of activated expert subnetworks, and reset the basic weights of the expert subnetworks that are not included in the set of activated expert subnetworks to 0. Alternatively, select a predetermined number of expert subnetworks from all expert subnetworks in the third-level group, sorted by their basic weights from largest to smallest, as the target expert subnetworks to obtain the set of activated expert subnetworks, and reset the basic weights of expert subnetworks not included in the set of activated expert subnetworks to 0.

[0047] The following example illustrates the processing of only activating the target expert subnetwork in the third-level group mentioned above, where the semantic relevance of the target attribute in the attribute constraint condition reaches the second threshold, using the formula.

[0048] For the third-level group mentioned above, in order to control the target attribute, the basic weights are obtained. Then, sparsification can be further performed. A sparsification threshold can be set. Alternatively, specify to retain the Top-k expert subnetworks. Construct the set of activated experts. For those not selected into the set The expert subnetwork has its weights forcibly set to 0. The final adjusted weights are used for parameter aggregation. Its expression is as follows: ; In this way, the higher-level parts of the model can automatically filter out expert subnetwork paths that are irrelevant to the target attribute in the current attribute constraints, and focus on the specific generation target.

[0049] In one or more embodiments of this disclosure, the weight increment of each expert subnetwork in each layer of the low-rank expert subnetwork group is decomposed into the product of a first low-rank matrix and a second low-rank matrix, wherein the first low-rank matrix is ​​a dimension-reduced matrix and the second low-rank matrix is ​​a dimension-increasing matrix.

[0050] In order to avoid updating the weights of the dense layers (dense layers refer to fully connected layers in the network where all input features are fully connected to the output features) of the pre-trained large language model Based on this, efficient fine-tuning of model parameters is achieved for each expert subnetwork in each layer. Constructing parallel low-rank adaptation (LoRA) branches can improve the adaptability and accuracy of large language models for specific tasks (such as multimodal fusion and position-controlled generation), while also considering model training and inference efficiency. For the first... The first layer Each expert subnetwork, with its weight increment It is decomposed into the product of two low-rank matrices: ; In the above formula, It is a dimensionality reduction matrix used to project a high-dimensional input into a low-dimensional latent space, and can be randomly initialized using a Gaussian distribution. It is an updimensional matrix used to map low-dimensional features back to the output space, and can be initialized as an all-zero matrix. It is the rank, a hyperparameter much smaller than the dimensions of the input and output (e.g., or This ensures that the number of training parameters is extremely small. Because... Initialize to zero, initial training phase This ensures that the model can start smoothly from its pre-trained initial state.

[0051] In one or more embodiments of this disclosure, the hierarchical fusion features are input into the low-rank expert subnetwork group of the large language model to obtain the target encoded features, which may specifically include: The weight increments of the target expert subnetworks are weighted and summed based on the basic weights of all target expert subnetworks in the set of activated expert subnetworks, and then multiplied by a preset scaling factor to obtain the aggregated weight increments. By weighted aggregating the weight increments, the model can dynamically adjust its parameter behavior for each layer and each token.

[0052] The aggregated weight increments are applied to the input hidden state of the current coding layer, and the output features of the current coding layer are calculated based on the matrix multiplication associative law. The output features are processed by layer normalization and residual connection, and then passed layer by layer as the input of the next layer encoder until the target encoded features are generated.

[0053] The following example illustrates the process of inputting hierarchical fusion features into the low-rank expert subnetwork group of a large language model to obtain target encoded features, using formulas as examples.

[0054] Optionally, the adjusted weights can be based on the above. and activation expert collection Calculate the final weight increment that takes effect in the current coding layer. This is a weighted summation process that aggregates only the parameters of the activated (non-zero weight) expert subnetworks, with the weight increments... The expression is as follows: ; In the above formula, This is a scaling factor used to adjust the influence of the low-rank matrix on the original model output (it can be set to...). This step transforms discrete routing decisions into continuous parameter tuning.

[0055] Optionally, the aggregated weight increments can be applied to the input hidden state of the current layer. Above, calculate the output. To reduce computational complexity, the actual operation can utilize the associative law of matrix multiplication, without explicitly constructing a huge matrix multiplication table. Instead of calculating matrices, the following order is used: ; In the above formula, first input Multiply Dimension reduction, then multiplication The dimensions are increased, and finally a weighted sum is performed. This calculation result... Then, it undergoes layer normalization and residual connection processing, serving as the input to the next coding layer, and is passed up layer by layer until the final hidden layer representation, i.e., the target encoded feature, is generated.

[0056] In one or more embodiments of this disclosure, the text generation method may further include: After obtaining the target text, a negative log-likelihood loss is calculated based on the multimodal fusion features and the target text. The target text is input into a pre-trained text attribute classifier with frozen parameters to obtain the predicted text attribute probabilities, and the binary cross-entropy loss is calculated based on the text attribute probabilities. The negative log-likelihood loss and the binary cross-entropy loss are weighted and fused together to obtain the total loss. The gradient of the total loss with respect to the key vector, the first low-rank matrix, and the second low-rank matrix is ​​calculated using the backpropagation algorithm. Freeze the original parameters of the large language model, and update the key vector, the first low-rank matrix, and the second low-rank matrix according to the gradient.

[0057] Specifically, updating the key vector, the first low-rank matrix, and the second low-rank matrix based on the gradient can be done by adjusting the update magnitude based on the gradient magnitude; the larger the absolute value of the gradient, the larger the parameter update magnitude.

[0058] During the training phase of the large language model, to ensure that the generated text is linguistically fluent, grammatically correct, and conforms to multimodal context logic, the model adopts a standard causal language modeling objective. Given a multimodal input context... (Including image features and cue words) and target response sequence The model needs to maximize the predicted probability of the next token. This involves generating a loss. Defined as negative log-likelihood loss: ; In the above formula, Is the model in the first The predicted probability distribution of each step. This loss enables the model to learn general language expression ability and multimodal understanding ability.

[0059] To force the text generated by the model to strictly adhere to preset attribute constraints, such as the target position (an example of the text attributes mentioned above) label. s ∈0,1 (e.g.) Representing "opposition", Representing "support", a pre-trained, parameter-frozen stance classifier can be introduced (an example of the text attribute classifier mentioned above). The current output distribution generated by the model is input into this classifier to obtain the predicted stance probabilities. (An example of the text attribute probabilities mentioned above). Stance Loss (As an example of the binary cross-entropy loss above) Calculated as binary cross-entropy loss: ; The binary cross-entropy loss term serves as a supervisory signal, penalizing the model for generating content that contradicts the target stance and guiding the model parameters to be updated in a direction that can generate the correct stance.

[0060] During the training phase, the two parts of the loss can be weighted and fused to form the final overall optimization objective, i.e., the total loss, which is expressed as follows: ; In the above formula, To balance the hyperparameters, the importance weights of the position control during training are adjusted. The total loss is calculated with respect to the LoRA parameters (which may include the routing key vector) using the backpropagation algorithm. and expert matrix The gradient of the language model is calculated, and only these newly added parameters are updated, while the original parameters of the large language model are kept frozen.

[0061] The embodiments of this disclosure, by jointly optimizing the text generation loss and the attribute classification loss, achieve significant controllability of text attributes while maintaining topic relevance and language naturalness, thereby enabling the generation of high-quality, targeted multimodal responses based on multimodal input.

[0062] During the inference phase, the model utilizes a pre-trained hierarchical routing strategy and a low-rank expert subnetwork group for generation. For a new input... The model calculates routing weights layer by layer and applies them to the expert subnetwork for updates. Finally, the top-level output Logits are decoded to obtain the final text response. The response It can simultaneously meet the relevance requirements of visual context and the constraints of user-specified text attributes.

[0063] The performance of the text generation method of this disclosure on the StanceGen2024 and MmMtCSD datasets for the multimodal stance-driven controllable statement generation task is shown in Table 1 below. The three evaluation metrics are: stance controllability (whether the generated response accurately conforms to the preset stance), topic relevance (whether the response is closely related to the text and image context), and perplexity (the fluency of the text). Regarding stance consistency, the text generation method of this disclosure performs excellently on both datasets, with an average score superior to other baseline solutions.

[0064] Table 1

[0065] The text generation method of this disclosure achieves significant results in terms of stance controllability. As shown in Table 1 above, the controllability scores of the text generation method of this disclosure on the two datasets are 0.7385 and 0.6016, respectively, which are significantly better than multimodal large models (such as GPT-4o) and traditional hybrid expert methods (MixLoRA). This indicates that the high-level sparse routing strategy of the encoder in the text generation method of this disclosure can effectively shield the interference of irrelevant information, allowing the model to focus on reasoning about specific stances, thereby accurately generating responses that conform to the preset stance (support or opposition).

[0066] The text generation method of this disclosure achieves an optimal balance between text quality and stance control. Although increasing control constraints in text generation tasks may sacrifice the naturalness of language (i.e., leading to an increase in perplexity PPL), the text generation method of this disclosure maintains the lowest perplexity (31.88 on the StanceGen2024 dataset, significantly lower than 52.85 on the GPT-4o dataset) while significantly improving controllability. This is due to the uniform mixing strategy at the bottom layer of the encoder in the text generation method of this disclosure, which ensures that the model can fully utilize all expert subnetworks to extract common linguistic features at the bottom layer, thereby guaranteeing the fluency and naturalness of the generated text.

[0067] In terms of relevance metrics, the text generation method of this disclosure also performs excellently, with metrics of 0.5587 and 0.7984, respectively, outperforming other multimodal baselines. Ablation experiments show that if the image input is removed, the model's controllability decreases from 0.7385 to 0.6921, and the relevance decreases from 0.5547 to 0.5346. This demonstrates that the mid-level weighted routing mechanism of the encoder in the text generation method of this disclosure can effectively fuse visual and textual semantics through an attention mechanism, utilizing visual information to assist in stance reasoning, thus solving the problem of effectively aligning image and text features in related technologies.

[0068] Compared to the variant that removes hierarchical dynamic routing (without HDR), the full model shows a significant performance improvement (controllability increased from 0.7187 to 0.7385). This demonstrates that the design approach of dividing the encoder network into "lower-level general, mid-level fusion, and high-level inference" layers and employing different routing strategies for each layer can better adapt to the deep semantic changes of large language models and is key to improving the overall performance of the model.

[0069] To further quantify the contribution of each key technical component in the text generation method of this disclosure to the model performance, ablation experiments were conducted on the StanceGen2024 dataset. The experimental results are shown in Table 2 below, comparing the performance of the complete model (HDRMLoE) with three variant models: removing image input (w / o Image), removing stance guidance loss (w / o L_stance), and removing hierarchical dynamic routing mechanism (w / o HDR).

[0070] Table 2

[0071] One or more embodiments of this disclosure also provide a computer device, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the one or more programs include instructions for a text generation method according to one or more embodiments of this disclosure.

[0072] One or more embodiments of this disclosure also provide a non-volatile computer-readable storage medium containing a computer program that, when executed by one or more processors, causes the one or more processors to perform a text generation method according to one or more embodiments of this disclosure.

[0073] This disclosure also provides a computer program product, including one or more computer programs, which, when executed by one or more processors, implement the text generation method of one or more embodiments of this disclosure.

[0074] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0075] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] For ease of description, the above computer devices are described in terms of function, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0077] The computer device described in the above embodiments is used to implement the corresponding text generation method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0078] This disclosure also provides a computer device for implementing the above-described text generation method. Figure 4 A schematic diagram of the hardware structure of an exemplary computer device 400 provided in an embodiment of this disclosure is shown. The computer device 400 can be used to implement... Figure 1 Server 106 can also be used to implement Figure 1 Terminal devices 102 and 104. In some scenarios, this computer device 400 can also be used to implement... Figure 1 Database server 108.

[0079] like Figure 4 As shown, the computer device 400 may include: a processor 402, a memory 404, a network interface 406, a peripheral interface 408, and a bus 410. The processor 402, memory 404, network interface 406, and peripheral interface 408 are interconnected within the computer device 400 via the bus 410.

[0080] Processor 402 may be a central processing unit (CPU), image processor, neural network processor (NPU), microcontroller (MCU), programmable logic device, digital signal processor (DSP), application-specific integrated circuit (ASIC), or one or more integrated circuits. Processor 402 can be used to perform functions related to the techniques described in this disclosure. In some embodiments, processor 402 may also include multiple processors integrated as a single logic component. For example, such as... Figure 4 As shown, processor 402 may include multiple processors 402a, 402b and 402c.

[0081] Memory 404 can be configured to store data (e.g., instructions, computer code, etc.). Figure 4As shown, the data stored in memory 404 may include program instructions (e.g., one or more programs for implementing the text generation method of embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). Processor 402 may also access the program instructions and data stored in memory 404 and execute the program instructions to operate on the data to be processed. Memory 404 may include volatile or non-volatile storage devices. In some embodiments, memory 404 may include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid-state drive (SSD), flash memory, memory stick, etc.

[0082] Network interface 406 can be configured to provide communication with other external devices to computer device 400 via a network. This network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the specific examples described above.

[0083] Peripheral interface 408 can be configured to connect computer device 400 to one or more peripheral devices to enable information input and output. For example, peripheral devices may include input devices such as keyboard, mouse, touchpad, touch screen, microphone, and various sensors, as well as output devices such as monitor, speaker, vibrator, and indicator lights.

[0084] Bus 410 can be configured to transfer information between various components of computer device 400 (e.g., processor 402, memory 404, network interface 406, and peripheral interface 408), such as internal buses (e.g., processor-memory bus), external buses (USB port, PCI-E bus), etc.

[0085] It should be noted that although the architecture of the computer device 400 described above only shows the processor 402, memory 404, network interface 406, peripheral interface 408, and bus 410, in specific implementations, the architecture of the computer device 400 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the architecture of the computer device 400 described above may only include the components necessary for implementing the embodiments of this disclosure, and does not necessarily include all the components shown in the figures.

[0086] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides a non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the one or more processors to perform the text generation method.

[0087] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0088] The computer program stored in the storage medium of the above embodiments is used to cause the one or more processors to execute the text generation method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0089] Based on the same inventive concept, corresponding to the text generation method of any of the above embodiments, this disclosure also provides a computer program product, which includes one or more computer programs. In some embodiments, the one or more computer programs are executable by one or more processors to cause the one or more processors to perform the text generation method. Corresponding to the execution entity for each step in each embodiment of the text generation method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0090] The computer program product of the above embodiments is used to cause the processor to execute the text generation method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0091] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0092] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0093] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0094] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A text generation method, characterized in that, include: Acquire image and text data, wherein the text data includes text attribute constraints; Initial features are extracted from the image and text data respectively to obtain initial image features and initial text features; The initial features of the image and the initial features of the text are mapped to a shared feature space of the same dimension to obtain aligned multimodal fusion features; The multimodal fusion features are input into the encoder of a pre-trained large language model. The encoder includes multiple coding layers, each of which includes at least two level groups. Routing weights for each level group are calculated based on a preset routing network corresponding to each level group. The level features output by each level group are weighted and fused according to the routing weights to obtain the level fusion features. The routing weights are used to characterize the importance of the features of the corresponding level group to the current coding task. The hierarchical fusion features are input into the low-rank expert subnetwork group of the large language model to obtain the target encoding features, wherein the low-rank expert subnetwork group includes at least two low-rank expert subnetworks. An embedding vector corresponding to the attribute constraints is constructed, and the target encoded features are fused with the embedding vector and then input into the decoder of the large language model to obtain the target text.

2. The method according to claim 1, characterized in that, The multi-level coding layer includes: a first-level group, a second-level group, and a third-level group. Routing weights for each level group are calculated based on a preset routing network corresponding to each level group. The hierarchical features output by each level group are weighted and fused according to the routing weights to obtain hierarchical fused features, including: Lexical and syntactic features of the multimodal fusion features are extracted based on the expert subnetworks in the first-level group, wherein the difference between the weight of each expert subnetwork in the first-level group and the average weight does not exceed a first threshold. Weights are assigned to the expert subnetworks in the second-level group based on the attention mechanism. The lexical features and the syntactic features are then weighted and fused according to the weights of each expert subnetwork in the second-level group to obtain the fused features. Weights are assigned to the expert subnetworks in the third-level group based on the attention mechanism. Only the target expert subnetworks in the third-level group whose semantic relevance to the target attribute in the attribute constraint reaches the second threshold are activated. The fusion features are processed based on the target expert subnetworks to obtain and output the hierarchical fusion features.

3. The method according to claim 2, characterized in that, Assigning weights to the expert subnetworks in the second-level group or the third-level group based on an attention mechanism includes: Project the input hidden state of the first coding layer containing the expert subnetwork into a routing query vector; Set the key vector for the first coding layer; Based on the routing query vector and the key vector, the routing score of the expert subnetwork is calculated using the dot product attention mechanism. The routing score is then normalized to obtain the basic weights of the expert subnetwork in the first coding layer.

4. The method according to claim 3, characterized in that, Only the target expert subnetwork in the third-level group whose semantic relevance to the target attribute in the attribute constraint reaches a second threshold is activated, including: Select the expert subnetworks whose basic weights are greater than the preset sparsity threshold as the target expert subnetworks to obtain the set of activated expert subnetworks, and reset the basic weights of the expert subnetworks that are not included in the set of activated expert subnetworks to 0. Alternatively, select a predetermined number of expert subnetworks from all expert subnetworks in the third-level group, sorted by their basic weights from largest to smallest, as the target expert subnetworks to obtain the set of activated expert subnetworks, and reset the basic weights of expert subnetworks not included in the set of activated expert subnetworks to 0.

5. The method according to claim 1, characterized in that, The weight increment of each expert subnetwork in each layer of the low-rank expert subnetwork group is decomposed into the product of a first low-rank matrix and a second low-rank matrix, wherein the first low-rank matrix is ​​a dimension-reduced matrix and the second low-rank matrix is ​​a dimension-increasing matrix.

6. The method according to claim 5, characterized in that, The hierarchical fusion features are input into the low-rank expert subnetwork group of the large language model to obtain the target encoded features, including: The weight increments of the target expert subnetwork are weighted and summed based on the basic weights of the target expert subnetwork, and then multiplied by a preset scaling factor to obtain the aggregated weight increments. The aggregated weight increments are applied to the input hidden state of the current coding layer, and the output features of the current coding layer are calculated based on the matrix multiplication associative law. The output features are processed by layer normalization and residual connection, and then passed layer by layer as the input of the next layer encoder until the target encoded features are generated.

7. The method according to claim 6, characterized in that, The method further includes: After obtaining the target text, a negative log-likelihood loss is calculated based on the multimodal fusion features and the target text. The target text is input into a pre-trained text attribute classifier with frozen parameters to obtain the predicted text attribute probabilities, and the binary cross-entropy loss is calculated based on the text attribute probabilities. The negative log-likelihood loss and the binary cross-entropy loss are weighted and fused to obtain the total loss. The gradient of the total loss with respect to the key vector, the first low-rank matrix, and the second low-rank matrix is ​​calculated using the backpropagation algorithm. Freeze the original parameters of the large language model, and update the key vector, the first low-rank matrix, and the second low-rank matrix according to the gradient.

8. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the one or more programs comprising instructions for performing the method of any one of claims 1 to 7.

9. A non-volatile computer-readable storage medium comprising a computer program, which, when executed by one or more processors, causes the one or more processors to perform the method of any one of claims 1 to 7.

10. A computer program product comprising one or more computer programs that, when executed by one or more processors, implement the method as described in any one of claims 1 to 7.

Images