Multimodal information generation and enhancement method based on AI large model

Abstract

The application relates to the technical field of multi-modal information processing, and discloses a multi-modal information generation and enhancement method based on an AI large model, which comprises the following steps: step 1, acquiring multi-modal original data and a task prompt, generating a unified token sequence and constructing a constraint graph; step 2, setting a modality quota, a coverage threshold and a consistency threshold according to a constraint node importance; step 3, generating a visibility mask and establishing a quota account book to control the visibility of dependent resources; step 4, determining a commitment section and generating an abstract fingerprint when a section meets a threshold; step 5, weighting candidate outputs according to the quota account book, and re-generating a mask when there is no feasible candidate; step 6, calculating a quality score, and performing backfill enhancement on a low-quality sub-section; and step 7, statistically calculating a modality conflict rate and a semantic drift rate, and adaptively adjusting the quota and the constraint weight. The application realizes semantic consistency maintenance, resource adaptive allocation and generation quality enhancement of a multi-modal information generation process.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information processing technology, specifically relating to a method for generating and enhancing multimodal information based on large AI models. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale AI models based on large-scale neural networks have made significant progress in fields such as natural language processing, computer vision, and speech recognition. In recent years, multimodal learning has become an important development direction for large-scale AI models. Its core goal is to integrate information from multiple modalities such as text, images, audio, and video to achieve cross-modal understanding and generation. However, existing multimodal generation technologies still have significant limitations: First, the feature spaces of different modalities are heterogeneous, making it difficult for models to achieve unified representation at the semantic level, leading to semantic drift and modal conflicts in the generated results; second, existing models mostly rely on static parameter configuration and lack dynamic adaptive adjustment mechanisms for modal weights, attention range, and quota resources during the generation process; third, the multi-turn semantic consistency and repetition suppression control in generation tasks are insufficient, often resulting in information redundancy, context breaks, or semantic imbalance in the models. Summary of the Invention

[0003] This invention provides a method for generating and enhancing multimodal information based on large AI models, which solves the technical problems of inconsistent semantic representations of different modalities, lack of dynamic constraint adjustment in the generation process, and difficulty in controlling semantic drift and repeated generation in multi-round generation.

[0004] This invention provides a method for generating and enhancing multimodal information based on large AI models, comprising the following steps:

[0005] Step 1: Obtain multimodal raw data and task prompts, discretize them to obtain a unified token sequence and label the modality; extract constraint requirements, construct a constraint graph, and bind dependent resources, effective windows, and target generation segments to the constraint nodes in the constraint graph, and record key fields;

[0006] Step 2: Set modal quotas, coverage thresholds, and consistency thresholds based on the importance of constraint nodes and context size to form a quota plan;

[0007] Step 3: Generate a visibility mask in the self-attention calculation, limit the attention scope of the unified token sequence based on the quota plan and establish a quota ledger, and turn off the visibility of the corresponding dependent resources when the modal quota is reached;

[0008] Step 4: When the target generated segment meets the generation coverage threshold and reaches the consistency threshold, determine the commitment segment, set the state cache of the commitment segment to read-only, and generate a digest fingerprint.

[0009] Step 5: In the autoregressive decoding, the candidate outputs are reweighted based on the quota ledger and the coverage status. If there are no feasible candidates, the visibility mask is adjusted within the uncommitted range to regenerate and update the quota ledger.

[0010] Step 6: Calculate the quality score, and perform localized backfill enhancement on sub-segments below the threshold within the corresponding valid window until the preset conditions are met;

[0011] Step 7: Calculate the modal conflict rate and semantic drift rate, and adaptively adjust the modal quota, effective window, and constraint weights based on the statistical results.

[0012] Furthermore, the generation of unified token sequences and the construction of constraint graphs include:

[0013] Step 11: Perform format consistency verification on the multimodal raw data, and convert each modal data into a corresponding discrete token sequence according to the discretizer of each modality, and concatenate them according to the preset sequence arrangement rules to obtain a unified token sequence.

[0014] Step 12: Parse and generate a set of constraint requirements according to the task prompts, and generate a constraint node for each constraint requirement; map the semantic targets of each constraint node based on the unified token sequence to form a correspondence between dependent resources, effective windows and target generation segments, and construct a constraint graph;

[0015] Step 13: Record a set of key fields for each constraint node. The set of key fields includes: object identifier, spatial coordinates, color label, and numerical entity.

[0016] Furthermore, the constraint nodes of the constraint graph represent a constraint target, and the edges include: dependency edges, referential edges, and exclusion edges. Among them, dependency edges are constructed based on semantic dependency analysis, referential edges are constructed based on object identifier matching, and exclusion edges are constructed based on task conflict rules.

[0017] Furthermore, based on the importance of constraint nodes and the context size, modal quotas, coverage thresholds, and consistency thresholds are set to form a quota plan, including:

[0018] Step 21: Based on the structural relationships of each constraint node in the constraint graph, the importance of the constraint node is obtained by weighting the dependency depth, the indicative association degree, and the key field complexity. The dependency depth represents the length of the longest directed path from the constraint node with no dependent incoming edges to the target constraint node. The indicative association degree represents the number of deduplicated adjacent nodes that have an indicative relationship with the target constraint node. The key field complexity represents the weighted scale of the various key fields of the constraint node.

[0019] Step 22: Calculate the modal weight distribution based on the token ratio of each modality in the dependent resources, and determine the modal quota of each constraint node in each modality by combining the corresponding number of dependent resources and the importance of constraint nodes, and set the coverage threshold and consistency threshold; wherein, by separately counting the number of tokens in the dependent resources of each modality and calculating the total number of tokens, the proportion of the number of tokens of each modality to the total number of tokens is calculated as the token ratio of that modality;

[0020] Step 23: Encapsulate the modal quota, coverage threshold, and consistency threshold of each constraint node into a quota plan structure, and generate a quota mapping table based on the node index and dependency relationship of the constraint graph to form a quota plan.

[0021] Furthermore, the generation of the visibility mask and the establishment of the quota ledger include:

[0022] Step 31: Construct a visibility mask matrix. Based on the modal quota, coverage threshold, and consistency threshold of each constraint node, filter the set of dependent resources and the effective window of the constraint node, and mark the dependent resources that meet the quota conditions as visible locations.

[0023] Step 32: Establish a quota ledger based on the quota plan. During the self-attention calculation process, use the visibility mask matrix to constrain the attention score. In multiple decoding steps, accumulate the accumulated consumption of each constrained node in different modalities in a time-weighted manner. When the accumulated attention of any modality reaches the modal quota threshold, close the visibility of the corresponding dependent resource.

[0024] Step 33: Perform quota state synchronization for constraint nodes with dependency or referential relationships. When the context dependency coverage ratio of the upstream node reaches the dependency coverage threshold, unlock the downstream node. When any node modality reaches the quota limit, synchronously turn off the visibility of the relevant node modality.

[0025] Furthermore, when the target generated segment meets the generation coverage threshold and reaches the consistency threshold, the committed segment is determined, the state cache of the committed segment is set to read-only, and a digest fingerprint is generated, including:

[0026] Step 41: Calculate the generation segment coverage ratio and semantic consistency score based on the target generation segment of each constraint node. The generation segment coverage ratio is the ratio of the number of generated tokens in the target generation segment to the preset total number of tokens. The semantic consistency score is the cosine similarity between the hidden vector of the target generation segment and the semantic center vector of the corresponding constraint node. When the generation segment coverage ratio reaches the generation coverage threshold and the semantic consistency score reaches the consistency threshold, the target generation segment is determined to meet the freezing condition.

[0027] Step 42: Perform a commitment freeze operation on the target generation segment that meets the freezing conditions. Set the corresponding constraint node status to commitment segment in the quota ledger, set the context cache of the generation segment to read-only, and block the gradient update of the generation segment in the self-attention calculation.

[0028] Step 43: Calculate the digest fingerprint for the frozen commitment segment. The digest fingerprint is calculated based on the average of all token hidden vectors in the commitment segment and converted into a fixed-dimensional fingerprint vector through a hash mapping function. When the similarity between the fingerprint vector of the newly generated segment and any digest fingerprint is higher than a preset similarity threshold, the generation of content that is semantically repetitive with the commitment segment is prohibited and a consistency check update is triggered.

[0029] Furthermore, in the autoregressive decoding, candidate outputs are reweighted based on the quota ledger and coverage status. When no feasible candidates are found, the visibility mask is adjusted within the uncommitted range to regenerate and update the quota ledger, including:

[0030] Step 51: Calculate the remaining quota ratio and coverage gap based on the constraint node status at the current decoding position. Perform exponential weighting on the candidate set according to the consistency of each modal quota, coverage threshold, and key fields in the quota ledger. The weighting coefficient is determined by the combination of the remaining quota ratio, coverage gap, and key field similarity according to a preset non-negative coefficient. If all candidates are masked by the visibility mask or the weighted probability is lower than the preset threshold, they are determined to be no feasible candidates.

[0031] Step 52: When no feasible candidate is found, backtrack to the nearest uncommitted position to perform local backfilling. Without crossing the boundaries of adjacent constraint nodes, minimize the translation of the local effective window within the preset maximum lookback step size, and recompile the visibility mask; regenerate the output based on the updated mask.

[0032] Step 53: After generating the output, update the cumulative consumption and coverage status of the corresponding constraint nodes in the quota ledger, disable the visibility of the dependent resources of the modal that has reached the quota limit, and propagate the update along the single-layer adjacent nodes of the dependent edge and the referential edge; when a circular dependency or conflict relationship is detected, terminate the propagation and roll back the state of the relevant node to the previous ledger version.

[0033] Furthermore, a quality score is calculated, and for segments below a threshold, localized backfill enhancement is performed within the corresponding valid window until a preset condition is met, including:

[0034] Step 61: The quality score of the token sequence in the target generated segment is obtained by weighting the semantic consistency score, modality matching degree and fluency probability. When any quality score is lower than the preset quality threshold, the position and its context window are marked as low quality sub-segment.

[0035] Step 62: While keeping the quota ledger and commitment frozen status unchanged, perform local backfill enhancement on the marked low-quality segments, generate a local visibility mask based on the window and re-decode it, use the enhanced sampling strategy to weight and update the candidate probability, and repeat the process until the enhanced quality score reaches the preset quality threshold.

[0036] Step 63: After completing the backfill enhancement, update the cumulative consumption and average quality records of the corresponding modality in the quota ledger, calculate the summary fingerprint of the enhancement content and perform consistency verification, and terminate the enhancement operation when the quality improvement rate is lower than the preset threshold.

[0037] Furthermore, after multiple rounds of enhancement, the summary fingerprints of all commitment segments are clustered to reduce redundant generation and enhance information fusion, including:

[0038] Step 71: Calculate the cosine similarity among all summary fingerprints, and group fingerprints with a cosine similarity higher than the clustering threshold into the same cluster;

[0039] Step 72: Weight the summary fingerprints within each cluster according to the semantic consistency score of the commitment segment, calculate the weighted semantic center vector, and form aggregated knowledge nodes;

[0040] Step 73: Feed the aggregated knowledge nodes back to the constraint graph to replace the original duplicate nodes, and merge the corresponding modal quotas and coverage thresholds in the quota ledger.

[0041] Furthermore, based on the statistical results, modal conflict rate and semantic drift rate are used to adaptively adjust modal quotas, effective windows, and constraint weights, including:

[0042] Step 81: In the target generation segment, the attention share of each modality is counted. Based on the condition that the difference between the attention shares of adjacent modalities is less than the conflict threshold and the maximum share is greater than the dominance threshold, a modality conflict event is determined. Based on the condition that the cosine similarity between the token hidden vector and the semantic center vector of the corresponding constraint node in the sliding window is less than the semantic drift threshold, a semantic drift event is determined. The modality conflict rate is obtained by counting the proportion of modality conflict events, and the semantic drift rate is obtained by counting the proportion of semantic drift events.

[0043] Step 82: When the modal conflict rate or semantic drift rate exceeds the set trigger threshold, calculate the quota adjustment amount, effective window adjustment amount, and constraint weight adjustment amount for each modality.

[0044] Step 83: Under the conditions of conservation constraints and smooth control, update the quotas of each modality according to the adjustment amount and normalize them proportionally, adjust the position and width of the corresponding effective window, and update the constraint weights; when the quality improvement rate after the update is lower than the preset improvement threshold or the modal conflict rate and semantic drift rate increase, roll back to the previous stable version.

[0045] The beneficial effects of this invention are as follows: By constructing a multimodal information generation and enhancement system based on a large AI model, this invention achieves unified representation, dynamic constraint control, and adaptive quality optimization of multimodal data; by establishing cross-modal semantic associations through unified token sequences and constraint graphs, it solves the semantic inconsistency problem caused by heterogeneous modal features in existing technologies; through visibility masks and quota ledger mechanisms, it precisely controls the attention range and modal resources during the generation process, avoiding conflicts and imbalances caused by intermodal competition; by introducing semantic consistency scores, quality scores, and backfilling enhancement strategies, it achieves self-detection and self-repair of generated content; and through feedback adjustment of modal conflict rates and semantic drift rates, it improves the stability and adaptability of the model. Overall, this invention improves the semantic consistency, information fusion, and generation quality of multimodal generation, while reducing the risks of redundant generation and semantic drift. Attached Figure Description

[0046] Figure 1 This is a flowchart of the multimodal information generation and enhancement method based on a large AI model according to the present invention. Detailed Implementation

[0047] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0048] like Figure 1 As shown, the method for generating and enhancing multimodal information based on large AI models includes the following steps:

[0049] Step 1: Obtain multimodal raw data and task prompts, discretize them to obtain a unified token sequence and label the modality; extract constraint requirements, construct a constraint graph, and bind dependent resources, effective windows, and target generation segments to the constraint nodes in the constraint graph, and record key fields;

[0050] Step 2: Set modal quotas, coverage thresholds, and consistency thresholds based on the importance of constraint nodes and context size to form a quota plan;

[0051] Step 3: Generate a visibility mask in the self-attention calculation, limit the attention scope of the unified token sequence based on the quota plan and establish a quota ledger, and turn off the visibility of the corresponding dependent resources when the modal quota is reached;

[0052] Step 4: When the target generated segment meets the generation coverage threshold and reaches the consistency threshold, determine the commitment segment, set the state cache of the commitment segment to read-only, and generate a digest fingerprint.

[0053] Step 5: In the autoregressive decoding, the candidate outputs are reweighted based on the quota ledger and the coverage status. If there are no feasible candidates, the visibility mask is adjusted within the uncommitted range to regenerate and update the quota ledger.

[0054] Step 6: Calculate the quality score, and perform localized backfill enhancement on sub-segments below the threshold within the corresponding valid window until the preset conditions are met;

[0055] Step 7: Calculate the modal conflict rate and semantic drift rate, and adaptively adjust the modal quota, effective window, and constraint weights based on the statistical results.

[0056] In one embodiment of the present invention, the generation of the unified token sequence and the construction of the constraint graph include:

[0057] Step 11: Perform format consistency verification on the multimodal raw data, and convert each modal data into a corresponding discrete token sequence according to the discretizer of each modality, and concatenate them according to the preset sequence arrangement rules to obtain a unified token sequence.

[0058] Specifically, the multimodal raw data includes text, images, audio, video, or other structured sensor data. The discretizer is a model component that converts continuous or unstructured data into discrete token sequences. For example, the text modality is encoded using a word-level segmenter, the image modality uses a visual encoder to convert image blocks into visual tokens, and the audio modality uses an acoustic front-end to extract feature spectrum frames and quantize them into acoustic tokens. Through discretization, data from different modalities are mapped to a unified symbol space, forming comparable discrete token representations within each modality. The sequence arrangement rules are determined based on the modal priority, temporal order, and contextual relationships specified in the task prompt. For example, for a task involving the joint generation of text and images, text tokens can be placed at the beginning of the sequence, and image tokens can be inserted at corresponding positions according to timestamps to maintain temporal and semantic alignment between modalities. Tokens for the text modality can be subwords or word fragments, tokens for the image modality can be visual feature blocks, and tokens for the audio modality can be acoustic feature vectors.

[0059] Step 12: Parse and generate a set of constraint requirements according to the task prompts, and generate a constraint node for each constraint requirement; map the semantic targets of each constraint node based on the unified token sequence to form a correspondence between dependent resources, effective windows and target generation segments, and construct a constraint graph;

[0060] Specifically, the task prompt is a natural language instruction or task description, which defines the type of generation target, constraints, and expected output modality. This step extracts the constraint information from the task prompt through semantic parsing and abstracts each constraint requirement into a constraint node. Each constraint node corresponds to a semantic target, such as a text description object, an image generation region, or an audio segment. Dependent resources are used to represent the preceding semantic content or context required for the node's generation; the effective window represents the effective decoding range of the node in the sequence, used to limit the visible area of ​​the attention mechanism; the target generation segment corresponds to the generation region of the node in the output sequence, used to control the modal position and content boundaries of the output.

[0061] The constraint nodes in the constraint graph represent a constraint target, and the edges include: dependency edges, referential edges, and exclusion edges; the constraint target refers to a semantic unit that needs to be generated or controlled by the AI ​​large model in a multimodal task, such as a text description object, an image region, or a semantic event.

[0062] The construction of dependency edges is based on semantic dependency analysis. For example, in text generation tasks, if a node describing an action depends on a preceding subject node, a directed dependency edge is formed in the graph, pointing from the subject node to the action node. The establishment of dependency edges ensures that the generation process follows logical temporal and semantic order, preventing logical breaks in the generated content.

[0063] The construction of referential edges is based on object identifier matching. For example, when instructions containing object names appear in different modal inputs, referential edges can associate these nodes with the same object semantic center, thereby achieving cross-modal semantic consistency maintenance.

[0064] The construction of exclusion edges is based on task conflict rules. For example, when two visual targets overlap in the same spatial coordinates, two action descriptions contradict each other, or two semantic instructions cannot be fulfilled at the same time, the system adds exclusion edges in the constraint graph to prevent the AI ​​large model from activating conflicting nodes at the same time during the generation process.

[0065] Step 13: Record a set of key fields for each constraint node. The set of key fields includes: object identifier, spatial coordinates, color label, and numerical entity. The object identifier is used to uniquely identify the target object, the spatial coordinates indicate the target's position in space or on the screen, the color label describes visual feature attributes, and the numerical entity represents the numerical semantic information involved in the task.

[0066] This embodiment achieves unified encoding, explicit dependency modeling, and controllable input generation for cross-modal data through the aforementioned unified token sequence and constraint graph construction process, ensuring that multimodal information maintains a consistent semantic constraint structure throughout the generation and enhancement process of large AI models.

[0067] In one embodiment of the present invention, a quota plan is formed by setting modal quotas, coverage thresholds, and consistency thresholds based on the importance of constraint nodes and the context size, including:

[0068] Step 21: Based on the structural relationships of each constraint node in the constraint graph, the importance of the constraint node is obtained by weighting the dependency depth, the indicative association degree, and the key field complexity. The dependency depth represents the length of the longest directed path from the constraint node with no dependent incoming edges to the target constraint node, which reflects the position of the node in the semantic dependency level. The indicative association degree represents the number of deduplicated adjacent nodes that have an indicative relationship with the target constraint node, which measures the association density of the node in the semantic indicative network. The key field complexity represents the weighted scale of the various key fields of the constraint node, which reflects the structural complexity of the node at the semantic description level.

[0069] Step 22: Calculate the modal weight distribution based on the token ratio of each modality in the dependent resources. Combined with the corresponding number of dependent resources and the importance of constraint nodes, determine the modal quota of each constraint node in each modality, and set a coverage threshold and a consistency threshold. The coverage threshold measures the semantic or spatial completeness of the generated content of the node, with a value ranging from 0.4 to 0.7. The consistency threshold measures the similarity between the generated result and the target semantic center, with a value ranging from 0.6 to 0.9. Specifically, the token ratio of each modality is calculated by counting the number of tokens in the dependent resources for each modality and calculating the total number of tokens. The modal quota reflects the attention resources or upper limit of the generation step size allocated to the node by the AI ​​large model during the generation process, and is used to control the distribution of computational focus of the model across different modalities.

[0070] Step 23: Encapsulate the modal quota, coverage threshold, and consistency threshold of each constraint node into a quota plan structure, and generate a quota mapping table based on the node index and dependency relationship of the constraint graph to form a quota plan.

[0071] This embodiment calculates the importance of constraint nodes, enabling the AI ​​large model to adaptively adjust the generation quotas of different modalities according to the task semantic structure, thereby achieving fine-grained scheduling of generation resources. By setting token ratios and thresholds, it ensures a balance in information content and semantic consistency of the generated content of each modality, avoiding generation offset caused by a single modality occupying too many computing resources. This realizes dynamic quota allocation and consistency control driven by the task semantic structure.

[0072] In one embodiment of the present invention, in a large AI model, a self-attention mechanism achieves the propagation of information associations within a sequence by calculating attention weights between different tokens. To avoid excessive focus on irrelevant or saturated resources, this embodiment filters and controls the set of dependent resources in the unified token sequence according to the aforementioned generated quota plan. The generation of the visibility mask and the establishment of the quota ledger include:

[0073] Step 31: Construct a visibility mask matrix. Based on the modal quota, coverage threshold, and consistency threshold of each constraint node, filter the dependent resource set and effective window of the constraint node. Mark dependent resources that meet the quota conditions as visible locations and dependent resources that do not meet the conditions as invisible locations. The rows of the visibility mask matrix represent query tokens, the columns represent key tokens, and the element values ​​are 1 or 0, representing visible and invisible, respectively. During self-attention calculation, the visibility mask matrix performs element-wise constraints on the attention weight matrix, so that the model can only focus on the resource locations marked as visible, thereby forming a structured attention visibility range during the generation process.

[0074] Step 32: Establish a quota ledger based on the quota plan. The quota ledger records the modal quota, coverage threshold, and consistency threshold of each constrained node under different modalities and initializes the state to active. During the self-attention calculation process, the attention score is constrained by the visibility mask matrix to obtain the effective attention weight after mask constraint. In multiple decoding steps, the accumulated consumption of each constrained node in different modalities is accumulated in a time-weighted manner to represent the proportion of quota consumed by the node in the current modality. When the accumulated attention of any modality reaches the modal quota threshold, the visibility of the corresponding dependent resource is turned off, and the visibility mark of the corresponding dependent resource is simultaneously modified to invisible.

[0075] It should be noted that the visibility mask is used for immediate attention constraints in the model computation layer, and the quota ledger is used for global state recording and dynamic threshold updates.

[0076] Step 33: Perform quota state synchronization on constraint nodes with dependency or referential relationships. Quota state synchronization is used to ensure the logical consistency of node states and the orderly propagation of dependencies within the constraint graph. When the context dependency coverage ratio of an upstream node reaches the dependency coverage threshold, the downstream node is unlocked, allowing it to participate in attention calculation and quota allocation in subsequent generation steps. When any node modality reaches the quota limit, the system synchronously disables the visibility of the relevant node modality along the dependency or referential edges of the constraint graph. The context dependency coverage ratio represents the actual coverage of dependent resources by the current node in the generation sequence, obtained by normalizing and summing the effective attention weights under the visibility mask constraint. The quota limit refers to the maximum attention consumption set for this node in the quota plan.

[0077] In one embodiment of the present invention, when the target generated segment meets the generation coverage threshold and reaches the consistency threshold, a commitment segment is determined, the state cache of the commitment segment is set to read-only, and a digest fingerprint is generated, including:

[0078] Step 41: Calculate the generation segment coverage ratio and semantic consistency score based on the target generation segments of each constraint node. The generation segment coverage ratio is the ratio of the number of generated tokens in the target generation segment to the preset total number of tokens, used to measure the model's generation progress on that target. The semantic consistency score is the cosine similarity between the hidden vector of the target generation segment and the semantic center vector of the corresponding constraint node, used to measure the semantic closeness between the generation segment and the semantic target of the target constraint node. When the generation segment coverage ratio reaches the generation coverage threshold and the semantic consistency score reaches the consistency threshold, the target generation segment is determined to meet the freezing condition. The hidden vector is the representation vector of each token in the target generation segment in the model's hidden layer space, and the semantic center vector is the feature center vector of the corresponding constraint node obtained by weighted aggregation in the multimodal semantic embedding space. By calculating the cosine similarity between the two, the consistency degree between the target generation segment and the semantic target of the constraint node can be obtained.

[0079] Step 42: Perform a commitment freeze operation on the target generation segment that meets the freezing conditions. Set the corresponding constraint node state to a commitment segment in the quota ledger and set the context cache of the generation segment to a read-only state. At the same time, block the gradient update of the generation segment in the self-attention calculation. That is, set the gradient of the frozen segment to zero in the backpropagation stage to ensure that its generation content remains stable in subsequent iterations and is no longer disturbed by the attention mechanism of subsequent modalities or nodes.

[0080] Step 43: Calculate the digest fingerprint for the frozen commitment segment. The digest fingerprint uniquely identifies the semantic features of a generated segment and is a compressed representation of the generated content in the semantic space. The digest fingerprint is calculated based on the mean of all token hidden vectors within the commitment segment and converted into a fixed-dimensional fingerprint vector through a hash mapping function. When the similarity between the fingerprint vector of a newly generated segment and any digest fingerprint is higher than a preset similarity threshold, the generation of content that semantically duplicates that of the commitment segment is prohibited, and a consistency check update is triggered to maintain the coherence and uniqueness of the global semantics. The hash mapping function can be a semantic hashing algorithm.

[0081] This embodiment ensures that the model freezes the corresponding generated segment in a timely manner after completing the target semantics by jointly determining the coverage ratio and semantic consistency score, preventing semantic drift caused by continued model iteration; through the summary fingerprint mechanism, the uniqueness identification and duplication suppression of the generated results are realized, avoiding the AI ​​large model from repeatedly generating the same semantic content in long sequences or multi-round tasks, realizing dynamic convergence and duplication suppression of multimodal generation tasks, and ensuring the semantic consistency of the generated results and the generation efficiency of the system.

[0082] In one embodiment of the present invention, in autoregressive decoding, candidate outputs are reweighted based on the quota ledger and coverage status. When no feasible candidates are found, the visibility mask is adjusted within the uncommitted range to regenerate and update the quota ledger, including:

[0083] Step 51: Calculate the remaining quota ratio and coverage gap based on the constraint node status at the current decoding position. Perform an exponential weighting on the candidate set according to the consistency of each modal quota, coverage threshold, and key fields in the quota ledger. The weighting coefficient is determined by a preset non-negative coefficient combination of the remaining quota ratio, coverage gap, and key field similarity. If all candidates are masked by the visibility mask or their weighted probability is lower than a preset threshold, they are determined to be non-feasible candidates. The remaining quota ratio is obtained by calculating the difference between the modal quota of the constraint node in the current modality and the accumulated consumption amount, and the ratio of the modal quota; it is set to zero when less than zero. The coverage gap is the difference between the dependent coverage threshold and the coverage ratio of the current position to the dependent resources; it is set to zero when less than zero. The coverage ratio is obtained by summing the dependent resources according to the effective attention weights after the visibility mask constraint and normalizing them.

[0084] Step 52: When no feasible candidate is found, backtrack to the nearest uncommitted position to perform local backfilling. Without crossing the boundaries of adjacent constraint nodes, minimize the translation of the local effective window within the preset maximum lookback step size to restore the context consistency of the position. Recompile the visibility mask and update the visible and invisible resource mapping relationship. Regenerate the output based on the updated mask.

[0085] Step 53: After generating the output, update the cumulative consumption and coverage status of the corresponding constraint nodes in the quota ledger. For modalities that have reached the quota limit, disable the visibility of their dependent resources and propagate the update along the single-layer adjacent nodes of the dependent edges and referential edges. When a circular dependency or conflict relationship is detected, terminate the propagation and roll back the status of the relevant nodes to the previous ledger version to ensure global consistency and logical closure of the ledger.

[0086] This embodiment uses the remaining quota ratio and coverage gap to jointly regulate the output probability, avoiding invalid outputs when the model is depleted or coverage is insufficient. Through local backfilling and re-decoding mechanisms, dynamic repair of generation breakpoints is achieved, making generation more continuous and realizing integrated closed-loop control of generation control, resource allocation, context repair and state management.

[0087] In one embodiment of the present invention, a quality score is calculated, and for segments below a threshold, localized backfill enhancement is performed within the corresponding valid window until a preset condition is met, including:

[0088] Step 61: The quality score of the token sequence within the target generated segment is obtained by weighting the semantic consistency score, modality matching degree, and fluency probability. When any quality score is lower than the preset quality threshold, the position and its context window are marked as a low-quality sub-segment. The semantic consistency score is used to evaluate the matching degree between the generated segment and the semantic target of the corresponding constraint node, and is calculated based on the cosine similarity between the hidden vector of the generated segment and the semantic center vector of the node. The modality matching degree represents the feature alignment degree of the generated segment across different modalities, reflecting the consistency of the generated content across modalities, and is obtained by calculating the weighted cosine similarity between the embedding vectors of text, image, audio, and other modalities. The fluency probability is a language fluency index calculated by the AI ​​large model in the decoding stage, representing the coherence and reasonableness of the generated sequence. It is calculated based on the token conditional probability output by the AI ​​large model in autoregressive decoding, and then normalized.

[0089] Step 62: While keeping the quota ledger and commitment frozen status unchanged, perform local backfill enhancement on the marked low-quality segments. Generate a local visibility mask based on the window and re-decode it. Limit the model to access only the context information related to the segment during backfill decoding to avoid cross-segment interference. Use the enhanced sampling strategy to update the candidate probability with weight and repeat until the enhanced quality score reaches the preset quality threshold.

[0090] Step 63: After completing the backfill enhancement, update the cumulative consumption and average quality records of the corresponding modality in the quota ledger, calculate the summary fingerprint of the enhanced content and perform consistency verification to ensure that the enhancement process does not introduce semantic conflicts or redundant duplications. When the quality improvement rate is lower than the preset threshold, terminate the enhancement operation.

[0091] It should be noted that the backfill enhancement in this embodiment is only performed on areas that are not committed or frozen, while frozen areas are in a read-only state.

[0092] This embodiment achieves quantitative evaluation of multimodal generation results through joint weighting of semantic consistency, modality matching degree, and fluency probability; improves local generation quality without affecting the global semantic structure through local backfilling and local decoding mechanisms; and introduces a summary fingerprint verification mechanism to effectively prevent semantic redundancy or duplicate output during the enhancement process, enabling the AI ​​large model to have self-evaluation and self-optimization capabilities during multimodal generation.

[0093] In one embodiment of the present invention, after multiple rounds of enhancement are completed, the summary fingerprints of all commitment segments are clustered to reduce redundant generation and enhance information fusion, including:

[0094] Step 71: Calculate the cosine similarity among all summary fingerprints, and group fingerprints with a cosine similarity higher than the clustering threshold into the same cluster; the summary fingerprint is a compressed representation of the commitment segment in the semantic space, reflecting the core semantic features of the segment; the clustering threshold ranges from 0.05 to 0.2.

[0095] Step 72: Weight the summary fingerprints within each cluster according to the semantic consistency score of the commitment segment, calculate the weighted semantic center vector, and form aggregated knowledge nodes; this step enables the AI ​​large model to automatically merge duplicate or similar content, and achieve semantic-level information compression and knowledge integration.

[0096] Step 73: Feed the aggregated knowledge nodes back to the constraint graph to replace the original duplicate nodes, thereby reducing redundant information in the graph structure and improving the compactness of semantic connections. In the quota ledger, merge the corresponding modal quotas and coverage thresholds. The merging rule is: take the weighted average of the sum of the modal quotas of each merged node and the coverage threshold to update the ledger record.

[0097] This embodiment automatically detects and merges semantically similar commitment segments through summary fingerprint clustering, reducing the rate of repeated generation; by aggregating knowledge nodes and feeding them back to the constraint graph, the model has the ability to dynamically update knowledge and compress information. Overall, it realizes global knowledge integration, semantic compression and repetition suppression of multimodal generation results based on a large AI model.

[0098] In one embodiment of the present invention, the modal conflict rate and semantic drift rate are statistically analyzed, and modal quotas, effective windows, and constraint weights are adaptively adjusted based on the statistical results, including:

[0099] Step 81: In the target generation segment, the attention share of each modality is calculated. Based on the condition that the difference in attention share between adjacent modalities is less than the conflict threshold and the maximum share is greater than the dominance threshold, a modality conflict event is determined, that is, different modalities compete for the same generation target in the same context. Based on the condition that the cosine similarity between the token hidden vector and the semantic center vector of the corresponding constraint node in the sliding window is less than the semantic drift threshold, a semantic drift event is determined, that is, the generated content deviates from the original semantic target. The modality conflict rate is obtained by calculating the proportion of modality conflict events, and the semantic drift rate is obtained by calculating the proportion of semantic drift events. The attention share represents the proportion of the token of that modality in the total attention weight. When the two events are triggered concurrently, the semantic drift rate is the dominant adjustment signal, and the modality conflict rate is used as an auxiliary factor for weighted correction.

[0100] Step 82: When the modal conflict rate or semantic drift rate exceeds the set trigger threshold, calculate the quota adjustment amount, effective window adjustment amount, and constraint weight adjustment amount for each modality; wherein, the quota adjustment amount is obtained by weighting the semantic drift rate, modal conflict rate, and the event correlation coefficient of the corresponding modality; the effective window adjustment amount is used to fine-tune the window position and width according to the drift rate and conflict rate; and the constraint weight adjustment amount determines the adjustment direction according to the difference between the drift rate and the conflict rate.

[0101] Step 83: Under the conditions of conservation constraints and smoothing control, update the quotas of each modality according to the adjustment amount and normalize them proportionally, adjust the corresponding effective window position and width, and update the constraint weights; when the quality improvement rate after the update is lower than the preset improvement threshold or the modality conflict rate and semantic drift rate increase, roll back to the previous stable version and reduce the smoothing coefficient. Conservation constraints mean that the total quota of each modality remains constant after the update to avoid global generation resource imbalance; smoothing control means that a smoothing coefficient is introduced during the update process to prevent parameter oscillations or abrupt changes.

[0102] This embodiment dynamically adjusts modal resource allocation through joint feedback of modal conflict rate and semantic drift rate to prevent attention monopoly caused by excessive strength of a single modality; semantic drift detection ensures that the generated content always converges around the semantic goal of the constraint node, avoiding generation deviation and semantic collapse, so that the AI ​​large model has the ability to monitor, optimize dynamically and automatically balance in the process of multimodal information generation.

[0103] It should be noted that the interval and threshold sizes are set for ease of comparison. The size of the threshold depends on the amount of sample data and the base number set by those skilled in the art for each set of sample data, as long as it does not affect the proportional relationship between the parameter and the quantized value. Furthermore, the above formulas are all dimensionless calculations, and the formulas are derived from software simulations using a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0104] The embodiments of the present invention have been described above, but the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of the present embodiments, all of which are within the protection scope of the present embodiments.

Claims

1. A method for generating and enhancing multimodal information based on large AI models, characterized in that, Includes the following steps: Step 1: Obtain the original multimodal data and task prompts, discretize them to obtain a unified token sequence and label the modalities; Extract constraint requirements, construct a constraint graph, and bind dependent resources, effective windows, and target generation segments to constraint nodes in the constraint graph, and record key fields; among them, multimodal raw data includes text, images, audio, video or other structured sensor data; Step 2: Set modal quotas, coverage thresholds, and consistency thresholds based on the importance of constraint nodes and context size to form a quota plan; Step 3: Generate a visibility mask during self-attention computation, limit the scope of attention for the unified token sequence based on the quota plan, and establish a quota ledger. When the modal quota is reached, disable the visibility of the corresponding dependent resources, including: Step 31: Construct a visibility mask matrix. Based on the modal quota, coverage threshold, and consistency threshold of each constraint node, filter the dependent resource set and effective window of the constraint node. Mark dependent resources that meet the quota conditions as visible locations and dependent resources that do not meet the conditions as invisible locations. The rows of the visibility mask matrix represent query tokens, the columns represent key tokens, and the element values ​​are 1 or 0, representing visible and invisible, respectively. During self-attention calculation, the visibility mask matrix performs element-wise constraints on the attention weight matrix, so that the model can only focus on the resource locations marked as visible. Step 32: Establish a quota ledger based on the quota plan. During the self-attention calculation process, use the visibility mask matrix to constrain the attention score. In multiple decoding steps, accumulate the accumulated consumption of each constrained node in different modalities in a time-weighted manner. When the accumulated attention of any modality reaches the modal quota threshold, close the visibility of the corresponding dependent resource. Step 33: Perform quota state synchronization for constraint nodes with dependency or referential relationships. When the context dependency coverage ratio of the upstream node reaches the dependency coverage threshold, unlock the downstream node. When any node modality reaches the quota limit, synchronously turn off the visibility of the relevant node modality. Step 4: When the target generated segment meets the generation coverage threshold and reaches the consistency threshold, determine the commitment segment, set the state cache of the commitment segment to read-only, and generate a digest fingerprint. Step 5: In the autoregressive decoding, the candidate outputs are reweighted based on the quota ledger and the coverage status. If there are no feasible candidates, the visibility mask is adjusted within the uncommitted range to regenerate and update the quota ledger. Step 6: Calculate the quality score, and perform localized backfill enhancement on sub-segments below the threshold within the corresponding valid window until the preset conditions are met; Step 7: Calculate the modal conflict rate and semantic drift rate, and adaptively adjust the modal quota, effective window, and constraint weights based on the statistical results.

2. The method for generating and enhancing multimodal information based on a large AI model according to claim 1, characterized in that, The generation of unified token sequences and the construction of constraint graphs include: Step 11: Perform format consistency verification on the multimodal raw data, and convert each modal data into a corresponding discrete token sequence according to the discretizer of each modality, and concatenate them according to the preset sequence arrangement rules to obtain a unified token sequence. Step 12: Parse and generate a set of constraint requirements according to the task prompts, and generate a constraint node for each constraint requirement; map the semantic targets of each constraint node based on the unified token sequence to form a correspondence between dependent resources, effective windows and target generation segments, and construct a constraint graph; Step 13: Record a set of key fields for each constraint node. The set of key fields includes: object identifier, spatial coordinates, color label, and numerical entity.

3. The method for generating and enhancing multimodal information based on a large AI model according to claim 1, characterized in that, The constraint nodes of the constraint graph represent a constraint target, and the edges include: dependency edges, referential edges, and exclusion edges. Among them, dependency edges are constructed based on semantic dependency analysis, referential edges are constructed based on object identifier matching, and exclusion edges are constructed based on task conflict rules.

4. The method for generating and enhancing multimodal information based on a large AI model according to claim 1, characterized in that, Based on the importance of constraint nodes and the context size, modal quotas, coverage thresholds, and consistency thresholds are set to form a quota plan, including: Step 21: Based on the structural relationships of each constraint node in the constraint graph, the importance of the constraint node is obtained by weighting the dependency depth, the indicative association degree, and the key field complexity. The dependency depth represents the length of the longest directed path from the constraint node with no dependent incoming edges to the target constraint node. The indicative association degree represents the number of deduplicated adjacent nodes that have an indicative relationship with the target constraint node. The key field complexity represents the weighted scale of the various key fields of the constraint node. Step 22: Calculate the modal weight distribution based on the token ratio of each modality in the dependent resources, and determine the modal quota of each constraint node in each modality by combining the corresponding number of dependent resources and the importance of constraint nodes, and set the coverage threshold and consistency threshold; wherein, by separately counting the number of tokens in the dependent resources of each modality and calculating the total number of tokens, the proportion of the number of tokens of each modality to the total number of tokens is calculated as the token ratio of that modality; Step 23: Encapsulate the modal quota, coverage threshold, and consistency threshold of each constraint node into a quota plan structure, and generate a quota mapping table based on the node index and dependency relationship of the constraint graph to form a quota plan.

5. The method for generating and enhancing multimodal information based on a large AI model according to claim 1, characterized in that, When the target generated segment meets the generation coverage threshold and reaches the consistency threshold, the committed segment is determined, the state cache of the committed segment is set to read-only, and a digest fingerprint is generated, including: Step 41: Calculate the generation segment coverage ratio and semantic consistency score based on the target generation segment of each constraint node. The generation segment coverage ratio is the ratio of the number of generated tokens in the target generation segment to the preset total number of tokens. The semantic consistency score is the cosine similarity between the hidden vector of the target generation segment and the semantic center vector of the corresponding constraint node. When the generation segment coverage ratio reaches the generation coverage threshold and the semantic consistency score reaches the consistency threshold, the target generation segment is determined to meet the freezing condition. Step 42: Perform a commitment freeze operation on the target generation segment that meets the freezing conditions. Set the corresponding constraint node status to commitment segment in the quota ledger, set the context cache of the generation segment to read-only, and block the gradient update of the generation segment in the self-attention calculation. Step 43: Calculate the digest fingerprint for the frozen commitment segment. The digest fingerprint is calculated based on the average of all token hidden vectors in the commitment segment and converted into a fixed-dimensional fingerprint vector through a hash mapping function. When the similarity between the fingerprint vector of the newly generated segment and any digest fingerprint is higher than a preset similarity threshold, the generation of content that is semantically repetitive with the commitment segment is prohibited and a consistency check update is triggered.

6. The method for generating and enhancing multimodal information based on a large AI model according to claim 1, characterized in that, In autoregressive decoding, candidate outputs are reweighted based on the quota ledger and coverage status. When no feasible candidates are found, the visibility mask is adjusted within the uncommitted range, and the quota ledger is regenerated and updated, including: Step 51: Calculate the remaining quota ratio and coverage gap based on the constraint node status at the current decoding position. Perform exponential weighting on the candidate set according to the consistency of each modal quota, coverage threshold, and key fields in the quota ledger. The weighting coefficient is determined by the combination of the remaining quota ratio, coverage gap, and key field similarity according to a preset non-negative coefficient. If all candidates are masked by the visibility mask or the weighted probability is lower than the preset threshold, they are determined to be no feasible candidates. Step 52: When no feasible candidate is found, backtrack to the nearest uncommitted position to perform local backfilling. Without crossing the boundaries of adjacent constraint nodes, minimize the translation of the local effective window within the preset maximum lookback step size, and recompile the visibility mask; regenerate the output based on the updated mask. Step 53: After generating the output, update the cumulative consumption and coverage status of the corresponding constraint nodes in the quota ledger, disable the visibility of the dependent resources of the modal that has reached the quota limit, and propagate the update along the single-layer adjacent nodes of the dependent edge and the referential edge; when a circular dependency or conflict relationship is detected, terminate the propagation and roll back the state of the relevant node to the previous ledger version.

7. The method for generating and enhancing multimodal information based on a large AI model according to claim 1, characterized in that, Calculate the quality score, and perform localized backfill enhancement on segments below the threshold within the corresponding valid window until the preset conditions are met, including: Step 61: The quality score of the token sequence in the target generated segment is obtained by weighting the semantic consistency score, modality matching degree and fluency probability. When the quality score of any token is lower than the preset quality threshold, the position of the token and its context window are marked as low quality sub-segment. Step 62: While keeping the quota ledger and commitment frozen status unchanged, perform local backfill enhancement on the marked low-quality segments, generate a local visibility mask based on the window and re-decode it, use the enhanced sampling strategy to weight and update the candidate probability, and repeat the process until the enhanced quality score reaches the preset quality threshold. Step 63: After completing the backfill enhancement, update the cumulative consumption and average quality records of the corresponding modality in the quota ledger, calculate the summary fingerprint of the enhancement content and perform consistency verification, and terminate the enhancement operation when the quality improvement rate is lower than the preset threshold.

8. The method for generating and enhancing multimodal information based on a large AI model according to claim 1, characterized in that, After multiple rounds of enhancement, the summary fingerprints of all commitment segments are clustered to reduce redundant generation and enhance information fusion, including: Step 71: Calculate the cosine similarity among all summary fingerprints, and group fingerprints with a cosine similarity higher than the clustering threshold into the same cluster; Step 72: Weight the summary fingerprints within each cluster according to the semantic consistency score of the commitment segment, calculate the weighted semantic center vector, and form aggregated knowledge nodes; Step 73: Feed the aggregated knowledge nodes back to the constraint graph to replace the original duplicate nodes, and merge the corresponding modal quotas and coverage thresholds in the quota ledger.

9. The method for generating and enhancing multimodal information based on a large AI model according to claim 1, characterized in that, The statistical modal conflict rate and semantic drift rate are used to adaptively adjust modal quotas, effective windows, and constraint weights based on statistical results, including: Step 81: In the target generation segment, the attention share of each modality is counted. Based on the condition that the difference between the attention shares of adjacent modalities is less than the conflict threshold and the maximum share is greater than the dominance threshold, a modality conflict event is determined. Based on the condition that the cosine similarity between the token hidden vector and the semantic center vector of the corresponding constraint node in the sliding window is less than the semantic drift threshold, a semantic drift event is determined. The modality conflict rate is obtained by counting the proportion of modality conflict events, and the semantic drift rate is obtained by counting the proportion of semantic drift events. Step 82: When the modal conflict rate or semantic drift rate exceeds the set trigger threshold, calculate the quota adjustment amount, effective window adjustment amount, and constraint weight adjustment amount for each modality. Step 83: Under the conditions of conservation constraints and smooth control, update the quotas of each modality according to the adjustment amount and normalize them proportionally, adjust the position and width of the corresponding effective window, and update the constraint weights; when the quality improvement rate after the update is lower than the preset improvement threshold or the modal conflict rate and semantic drift rate increase, roll back to the previous stable version.

Images