Reinforcement learning-based llm output stability control method and system

Abstract

The application provides an LLM output stability control method and system based on reinforcement learning, which comprises: multi-modal structural analysis of literature data, extraction of content units and construction of a feature vector set and a correlation mapping table; parallel driving of a text logic processing module and a visual semantic processing module to interpret cross-modal task units and construct an initial summary set; in an iterative closed loop driven by cross-modal consistency rewards, a joint action is constructed according to a current joint state, a reward signal is calculated by a cross-modal consistency arbitrator, a near-end policy optimization algorithm is used to update a policy network, and the process is repeated until the reward signal converges; based on the correlation mapping table, a sequence-to-sequence language model is driven to reconstruct a final text summary and an information description set of a final joint state set, and a final summary is generated. Through structural analysis and multi-modal collaborative control, the application ensures the consistency and stability of the summary content and is suitable for the fields of natural language processing and reinforcement learning.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and reinforcement learning, and particularly to a method and system for controlling the output stability of LLM based on reinforcement learning. Background Technology

[0002] With the widespread application of large-scale language models (LLMs) in fields such as intelligent analysis and knowledge discovery of scientific and technological literature, how to effectively control a single end-to-end language model to generate summaries that are both faithful to the original text and maintain logical consistency across cross-modal information when processing complex documents containing multimodal elements such as text, charts, and formulas has become a key technical challenge in the process of moving from single-point content generation to system-level stability control of large language models.

[0003] Chinese patent application CN119884355B discloses an LLM text processing method, which includes: acquiring input question information; inputting the input question information into a word segmentation database to output question words; inputting the question words into a preset vocabulary database to output synonyms, mutually exclusive words, and abbreviations, and defining the question words, synonyms, mutually exclusive words, and abbreviations as query words; outputting final question information based on the query words; using a preliminary detection method to find baseline answer information and its corresponding path based on the final question information; inputting the baseline answer information and path into a priority database to match priorities; determining preliminary detection answer information from the matched priorities; using a fine detection method to find fine detection answer information based on the preliminary detection answer information; and uploading and displaying the fine detection answer information as the final answer text.

[0004] However, current technology still faces many challenges. The bottleneck in the application of current large-scale language models lies primarily in the inherent consistency of handling cross-modal information. When processing critical scientific literature containing complex charts and graphs, such as clinical drug trial reports, models generally lack built-in mechanisms for cross-validating textual conclusions with chart data. This not only leads to deep semantic contradictions between the model's output textual conclusions and the quantitative data revealed by the charts, but also frequently results in deep semantic discrepancies. For example, the text summary might assert that a drug's efficacy is significant, while error bars in the chart indicate no statistically significant difference. This bias, which cannot be self-corrected by the model, renders the output summary unscientifically reliable, leading to a series of serious misjudgments. In basic research, this can lead to incorrect research directions and a huge waste of research resources; in critical fields such as biomedicine, misled summary conclusions could directly interfere with clinical treatment plans, posing a potential threat to patients' lives and health. Summary of the Invention

[0005] To achieve the above objectives, this invention provides an LLM output stability control method based on reinforcement learning, the specific technical solution of which is as follows:

[0006] Page layout analysis is performed on unstructured literature data to segment and extract multimodal content regions. The content regions are converted into a set of content units. Each content unit in the set of content units is vectorized to generate a set of feature vectors. Cross-modal association mapping between the content units is constructed to build an association mapping table. The literature data is then converted into a cross-modal literature dataset containing a set of content units, a set of feature vectors, and an association mapping table.

[0007] Based on the association mapping table, each non-text unit is formed into a text context set to construct a cross-modal task unit set. The text logic processing module and the visual semantic processing module are driven in parallel to perform preliminary interpretation of each cross-modal task unit, and output initial text summaries and initial information description sets respectively. The outputs of the two modules are aggregated to construct an initial summary set.

[0008] Initiate an iterative closed loop driven by cross-modal consistency rewards, construct joint actions based on the current joint state, calculate reward signals by a cross-modal consistency arbitrator based on the joint state, construct a proximal policy optimization algorithm based on the reward signals to update the policy networks of the text logic processing module and the visual semantic processing module, complete the state update, and determine whether to terminate the closed loop based on the convergence of the reward signals.

[0009] The final joint state set is deconstructed to separate the final text summary set and the final information description set. Using the association mapping table as a structured guide, the sequence-to-sequence language model is driven to fuse and reconstruct the two separated sets to generate the final summary.

[0010] Furthermore, the transformation steps of the cross-modal literature dataset include:

[0011] Page layout analysis and semantic segmentation are performed on unstructured literature data to identify and extract text, image and formula content areas with category labels and bounding boxes, and the extracted content areas are converted into a set of structured content units.

[0012] Based on the category label of each content unit in the content unit set, the corresponding text or visual encoder based on the Transformer architecture is called to encode the content unit into a corresponding feature vector, forming a feature vector set containing text, visual and formula semantics.

[0013] For the combination pairs of text block units and non-text units in the content unit set, a comprehensive association scoring algorithm is executed to calculate the explicit reference relationship score and the spatial proximity score respectively. The two scores are weighted to generate an association confidence score. The combination pairs with the association confidence score exceeding the preset confidence threshold are selected, and an association mapping table is constructed.

[0014] By executing all the above processes, a cross-modal literature dataset is obtained.

[0015] Furthermore, the method for constructing the initial summary set includes:

[0016] Based on the association mapping table, traverse each non-text unit in the content unit set, query and filter all associated text block units for the non-text unit to form a text context set, encapsulate each non-text unit and its corresponding text context set into a cross-modal task unit, and construct a cross-modal task unit set.

[0017] The driving text logic processing module performs deep semantic understanding and information compression on the text context set associated with each non-text unit, and generates an initial text summary representing the context set.

[0018] The visual semantic processing module is driven to perform visual feature extraction and semantic modeling on the pixel matrix of the visual object unit according to the category label of the non-text unit, and to perform syntax parsing and semantic understanding on the source code content of the formula object unit, generating the corresponding initial information description and constructing the initial information description set.

[0019] For each cross-modal task unit, the initial text summary set generated by the text logic processing module and the initial information description set generated by the visual semantic processing module are aggregated to construct an initial summary set.

[0020] Furthermore, the filtering method for the text context set includes having at least one triplet entry in the association mapping table, where each triplet entry has a corresponding text index and a corresponding non-text index. Based on this filtering method, a semantic association query operation is completed, extracting all text content that has a predetermined association with a given visual object unit.

[0021] Furthermore, the update steps for the policy networks of the text logic processing module and the visual semantic processing module include:

[0022] The text logic processing module and the visual semantic processing module are driven to input the local observations of the current joint state into their respective policy networks, generate probability distributions in the predefined action space, determine the corresponding actions based on the probability distributions, and construct a joint action by the actions determined by the two modules.

[0023] The cross-modal consistency arbitrator is driven to comprehensively evaluate the current joint state from three dimensions: semantic alignment reward, numerical consistency reward, and logical compliance reward. The reward signal is obtained by weighted summation of the reward scores of each dimension.

[0024] The advantage function is calculated based on the reward signal, and the objective function of the near-end policy optimization algorithm is constructed by combining the action probability ratio. The stochastic gradient ascent algorithm is used to synchronously update the parameter set of the text logic processing module and the parameter set of the policy network of the visual semantic processing module along the gradient direction of the objective function.

[0025] Furthermore, the semantic alignment reward is used to evaluate the degree of alignment between the text summary and the information description at the semantic level; the numerical consistency reward is used to verify whether the specific quantitative statements in the text description are consistent with the original non-text data; and the logical conformity reward is used to verify whether the structure, process, or logical relationship in the text description is consistent with the original non-text data.

[0026] Furthermore, the method for determining the termination of the closed loop includes: updating the current joint state based on the joint action to generate the joint state for the next iteration; monitoring the relative rate of change of the reward signal in the continuous iteration; if the absolute value of the relative rate of change is continuously lower than a preset convergence threshold, then terminating the iteration loop and outputting the final set of joint states; otherwise, feeding back the updated joint state to the decision-making step to start the next iteration.

[0027] Furthermore, the method for generating the final summary includes:

[0028] The final joint state set is reconstructed by extracting and separating the encapsulated text summary and information description to construct the final text summary set and the final information description set.

[0029] Using an association mapping table as a structured context guide, a sequence-to-sequence language model is driven to perform natural language generation tasks on the final text summary set and the final information description set to generate the final summary.

[0030] The LLM output stability control system based on reinforcement learning is used to implement the above-mentioned LLM output stability control method based on reinforcement learning. It includes a document modality segmentation module, a dual-module parallel interpretation module, a cross-modal reward arbitration module, and a joint state deconstruction module.

[0031] The document modality segmentation module is used to perform page layout analysis on unstructured document data to segment and extract multimodal content regions, convert the content regions into a set of content units, perform feature vectorization on each content unit in the set of content units to generate a set of feature vectors, and construct a cross-modal association mapping between the content units to build an association mapping table, thereby converting the document data into a cross-modal document dataset containing a set of content units, a set of feature vectors, and an association mapping table.

[0032] The dual-module parallel interpretation module is used to form a text context set of each non-text unit according to the association mapping table to construct a cross-modal task unit set, and to drive the text logic processing module and the visual semantic processing module in parallel to perform preliminary interpretation of each cross-modal task unit, respectively outputting an initial text summary and an initial information description set. The outputs of the two modules are aggregated to construct an initial summary set.

[0033] The cross-modal reward arbitration module is used to initiate an iterative closed loop driven by cross-modal consistency rewards, construct joint actions based on the current joint state, calculate reward signals by the cross-modal consistency arbitrator based on the joint state, construct a proximal policy optimization algorithm based on the reward signals to update the policy networks of the text logic processing module and the visual semantic processing module, complete the state update, and determine whether to terminate the closed loop based on the convergence of the reward signals.

[0034] The joint state deconstruction module is used to deconstruct the final joint state set to separate the final text summary set and the final information description set. Using the association mapping table as a structured guide, it drives the sequence-to-sequence language model to fuse and reconstruct the two separated sets to generate the final summary.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] This invention constructs an association mapping table that integrates explicit text reference relationships and physical spatial proximity, transforming the original implicit text-image correspondence in a document, which depends on the layout, into explicit structured associations that can be directly parsed by machines. This solves the problem of ambiguity and loss of cross-modal semantic associations caused by traditional models processing flat data streams and lacking structured prior knowledge.

[0037] This invention constructs a cross-modal task unit and drives two independent modules to perform parallel and decoupled interpretation. This decomposes the complex mixed modality understanding task into two highly focused single-modal analysis subtasks, providing clear and unbiased initial input for subsequent objective cross-modal consistency arbitration. This avoids the problems of inter-modal feature interference and semantic focus drift caused by premature information fusion in traditional single models.

[0038] This invention designs a multi-dimensional consistency reward mechanism that integrates macro-level semantic alignment, micro-level numerical verification, and deep logical conformity. This mechanism transforms the collaborative objective between two independent modules from a single, fuzzy semantic similarity problem into a quantifiable multi-objective optimization problem with hard constraints. This solves the problem of pseudo-alignment that often arises from existing technologies that rely solely on a single similarity index.

[0039] This invention uses an association mapping table as prior knowledge of the text structure to guide the generation of the final summary, transforming the process of splicing disordered fragments into a constrained fusion and reconstruction task. This solves the problem that traditional fusion methods, due to the lack of macro-structural information and the inability to perform only local text optimization, result in the break in the logical flow of the final summary and the loss of core image and text reference relationships. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the principle of the reinforcement learning-based LLM output stability control method of the present invention.

[0042] Figure 2 This is a functional block diagram of the LLM output stability control system based on reinforcement learning of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example 1

[0045] Please see Figure 1 As shown, this embodiment provides a reinforcement learning-based LLM output stability control method, including:

[0046] S1000, for unstructured literature data Perform page layout analysis to segment and extract multimodal content regions, and convert the content regions into sets of content units. U For the set of content units UEach content unit in the text is vectorized to generate a set of feature vectors. And construct cross-modal association mappings between the content units to build an association mapping table. M The document data Convert to a collection containing content units U eigenvector set F and associated mapping table M Cross-modal literature dataset .

[0047] Specifically, this step aims to convert the literature data stored in PDF format... Transforming unstructured formats that are human-readable but difficult for machines to parse directly into structured cross-modal literature datasets. By accurately segmenting and cross-modal associating multimodal elements such as text, icons, and formulas within the documents, data unification and machine readability are achieved, providing a reliable data foundation for subsequent stable and consistent abstract generation.

[0048] Further, step S1000 includes:

[0049] Step S1100, for unstructured document data Page layout analysis and semantic segmentation are performed to identify and extract text, image, and formula content regions with category labels and bounding boxes. The extracted content regions are then converted into a structured set of content units. .

[0050] Specifically, this step aims to transform the input, single, unstructured literature data... Decode and reconstruct it into a set of content units composed of discrete data objects. This achieves modal separation of the original document, that is, separating text, visual and formula content from the mixed layout of the page, and assigning clear category labels and physical location attributes to each separated object, so as to provide clear and non-interfering data input for subsequent steps.

[0051] In practice, this step is automatically executed by a document parsing engine deployed on the server. This document parsing engine transforms the input unstructured document data stream into a structured set of content units by executing a comprehensive document parsing function.

[0052] First, the input literature data Each page is rasterized, converting it from vector or hybrid formats into a pixel matrix representation easily processed by visual models. Subsequently, the document parsing engine's built-in deep neural network-based page layout analysis model, such as a Transformer-based LayoutLM variant architecture, analyzes this pixel matrix. This page layout analysis model identifies and locates content regions with independent functions within each page by performing pattern recognition algorithms such as semantic segmentation or object detection, and outputs a set of pages containing descriptions of multiple regions. Each region is labeled with its category. and bounding box coordinates Defined. Among them, This indicates the first page layout analysis model that has been identified. k The category labels for each region are limited to text (TEXT), image (IMAGE), or formula (FORMULA). Indicates the first k The bounding box of a region on the page is usually defined by a set of coordinate values ​​to describe the physical location and size of the region; k This represents the index of an independent content area within the page, used to traverse and uniquely identify the content area identified by the page layout analysis model. Its value ranges from 1 to the total number of areas identified on the current page.

[0053] Next, based on the category label Subsequent differentiated content extraction is performed on each content area, specifically:

[0054] First, for text regions categorized as TEXT, the Optical Character Recognition (OCR) algorithm is used to extract their digitized string content. The content of this string Its bounding box Together they form a text block unit .in, Indicates the first i The string content of each text block unit is a digital text sequence extracted by an OCR algorithm; i The sequence index of the text block unit, with a value ranging from 1 to the total number of text blocks; t A prefix identifier indicating the modality type, specifically referring to the text "Text", used to distinguish different modalities in variable naming; Indicates the first i The bounding box of each text block unit is used to describe the spatial position of the text block unit on the page; Indicates the first i A text block unit is a structured data object that encapsulates all the information of a single text block, containing the string content of that text block. and its bounding box .

[0055] Second, for visual regions categorized as IMAGE, their standardized pixel matrix is ​​obtained using a standard image decoding library. The pixel matrix Its bounding box Together they constitute a visual object unit .in, Indicates the first j The pixel data of each visual object unit is a standardized pixel matrix obtained by processing the standard decoding library and can be directly processed by computer vision models. j The sequence index of the visual object unit, with a value ranging from 1 to the total number of visual objects; v A prefix identifier indicating the modality type, specifically referring to Visual, used to distinguish different modalities in variable naming; Indicates the first j The bounding box of a visual object unit is used to describe the spatial position of the visual object on the page; Indicates the first j A visual object unit is a structured data object that encapsulates all the information of a single visual object, including the pixel data of that visual object. and its bounding box .

[0056] Third, for formula regions of type FORMULA, prioritize parsing their embedded structured source code such as MathML or LaTeX to obtain the source code content. If the parsing is successful, the source code content will be displayed. Will with its bounding box Together they form a single formula object unit. If parsing fails or the source code does not exist, the region will be treated as a special visual object, processed as an image, and included in the aforementioned visual object unit. In the set. Among them, Indicates the first Source code content of each formula object unit , is a text string describing the mathematical structure of the formula; The sequence index of the formula object cell, ranging from 1 to the total number of successfully parsed formula objects; e A prefix identifier indicating the modality type, specifically for Equation, used to distinguish different modalities in variable naming; Indicates the first The bounding box of each formula object cell is used to describe the spatial position of the formula object on the page; Indicates the first A formula object unit is a structured data object that encapsulates all the information of a single formula object, including the source code content of that formula object. and its bounding box .

[0057] Finally, by executing the above process, the document parsing function will process the entire document data. Completely map and transform into a structured collection of content units. This achieves a crucial transformation from unstructured data with a fixed format to structured multimodal content units, laying a solid data foundation for subsequent feature extraction and cross-modal correlation analysis. The content unit set... Composed of all text block units The set All visual object units The set and all formula object units The set Together they form a whole.

[0058] Step S1200, based on the content unit set The category labels of each content unit are assigned, and the corresponding text or visual encoder based on the Transformer architecture is invoked to encode the content unit into a corresponding feature vector, forming a feature vector set containing text, visual, and formula semantics. .

[0059] Specifically, this step aims to transform the discrete and heterogeneous set of content units produced in step S1100. It is mapped to a set of feature vectors consisting of high-dimensional real vectors through a set of deep learning models. This enables the numerical and semantic representation of content in different modalities.

[0060] In the specific implementation process, this step constructs and calls a set of pre-trained deep learning encoder models, which are implemented through a comprehensive feature extraction function, providing the input content unit set. U Each unit in the vector space is represented in a distributed manner in a high-dimensional vector space based on its modality type, as follows:

[0061] First, for text block units Invoke a text encoder based on the Transformer architecture. For example, the SciBERT model. The text encoder converts text blocks into units. The string content in As input, through multi-layer self-attention mechanism computation, the output is a text feature vector that can capture deep contextual semantics. .in, Indicates the first i text block unit The text feature vector is used to analyze text block units. The semantic information is numerically encoded.

[0062] Second, for visual object units Call a visual encoder For example, the ViT model based on Transformer. The visual encoder converts visual object units... pixel matrix As input, by extracting visual patterns from low-level textures to high-level abstractions layer by layer, a visual feature vector representing its visual content is output. .in, Indicates the first j visual object unit Visual feature vectors are used to analyze text block units. The visual patterns and structures are numerically encoded. It is worth noting that formula regions that failed to be parsed in step S1100 and were considered visual objects are also processed at this stage.

[0063] Third, for formula object units Using its source code content Essentially, it is a special sequence language with a strict syntax. The text encoder described above is reused. The process generates a formula feature vector that represents the mathematical structure and semantics of the expression. .in, Indicates the first Formula object unit The formula feature vector is used to define the formula object unit. The mathematical structure and semantics are numerically encoded.

[0064] Finally, by executing the above process, the feature extraction function extracts the entire set of content units. Completely transform into a unified and computable set of feature vectors. This completes the deep feature representation of the multimodal content of the document. The feature vector set... From all text feature vectors The set All visual feature vectors The set and all formula eigenvectors The set Together they form a whole.

[0065] Step S1300, for the content unit set U For pairs of text blocks and non-text blocks, a comprehensive association scoring algorithm is applied to calculate explicit reference relationship scores. Spatial proximity score The two scores are weighted to generate a correlation confidence score. Pairs of scores exceeding a preset confidence threshold are selected, and a correlation mapping table is constructed. M .

[0066] Specifically, this step aims to establish explicit semantic alignment between textual and non-textual content, for the discrete content unit set produced in step S1100. U Constructing a cross-modal quantitative association mapping table M The associated mapping table M As a core data structure, it is used to depict the correspondence and semantic dependencies between text block objects, visual object units, and formula object units, providing contextual constraints and correlation basis for consistency judgment and summary generation in subsequent collaborative work.

[0067] In the specific implementation process, this step targets the set of discrete content units. U Any text block unit in When combined with non-textual units, a comprehensive association scoring algorithm is applied to generate a quantified association confidence score. This comprehensive association scoring algorithm is implemented by weighted summation of two core indicators, as follows:

[0068] First, scoring of explicit reference relationships. This metric is used to evaluate text content. Does the text contain direct references to non-textual units? These non-textual units are visual object units. or formula object unit Through text block units string content It performs regular expression matching or named entity recognition to retrieve predefined keyword patterns. For example, for a text-visual pair, it retrieves "as shown in the figure". j "See table" j "etc. pattern; for text-formula pairs, search for "such as formula" "by formula" The pattern is "available". If there is an explicit reference, the score for this item is 1; otherwise, it is 0.

[0069] Second, spatial proximity scoring This metric is used to evaluate a collection of content units. U Arbitrary text block unit The degree of proximity of textual units to non-textual units on the physical layout of the document. Here, the non-textual units are visual object units. or formula object unit Based on the bounding box information of each unit obtained in step S1100, the Euclidean distance between the geometric center points of the text block unit and the non-text block unit is calculated, and this Euclidean distance is converted into a proximity score within the range of (0, 1) using a Gaussian kernel function. The closer the physical distance between the text unit and the non-text unit, the higher the score.

[0070] Scoring based on the above explicit reference relationships Spatial proximity score Two scoring metrics are used to calculate the association confidence score through a weighted summation. This association confidence score is the final quantitative basis for determining whether a strong semantic association exists between a text unit and a non-text unit. This calculation process is performed separately for the two cross-modal combinations, as follows:

[0071] First, for text block units With visual object unit Its association confidence score The calculation formula is as follows:

[0072] ;

[0073] in, Indicates the first i The text block unit and the first j The association confidence score between visual object units is a final evaluation value that combines explicit text references and spatial physical proximity to measure the strength of the association between the two. The weight hyperparameter is a floating-point number ranging from [0,1], used to balance the scoring of explicit referential relationships. Spatial proximity score The weights; Indicates the first i The text block unit and the first j The score for explicit reference relationships between visual object units is a binary value, either 0 or 1, used to characterize text block units. Does the content contain visual object units? Direct textual reference, such as "see figure". j ”; Indicates the first i The text block unit and the first j The spatial proximity score between visual object units is a real value in the range (0, 1], used to quantify the physical proximity of the two on the document page.

[0074] Second, for text block units With formula object unit Its association confidence score The calculation is as follows:

[0075] ;

[0076] in, Indicates the first i The text block unit and the first The association confidence score between formula object units is a final evaluation value that combines explicit text references and spatial physical proximity to measure the strength of the association between the two. Indicates the first i The text block unit and the first The score for explicit reference relationships between formula object units is a binary value, either 0 or 1, used to characterize text block units. Does the content contain formula object units? Direct textual reference, such as "as in the formula" As shown; Indicates the first i Each text block unit and The spatial proximity score between formula object units is a real value in the interval (0, 1], used to quantify the physical proximity of the two on the document page.

[0077] Finally, a mapping is constructed based on the calculated association confidence score. A predefined confidence threshold is set. The processor iterates through all text-non-text unit combinations, and if the calculated association confidence score is high enough, a mapping is constructed. or If the score is greater than the confidence threshold, then a strong correlation is determined between the two elements. The index of this unit combination pair and its corresponding correlation confidence score will be treated as a triple, i.e. or Stored in the final association mapping table. M .

[0078] Thus, by executing all steps S1100 to S1300 above, a complete and structured cross-modal literature dataset is obtained. This dataset is a composite data structure that encapsulates multi-level information, including a set of content units. eigenvector set and associated mapping table .

[0079] S2000, based on the association mapping table Each non-text unit is grouped into a text context set to construct a cross-modal task unit set. The parallel-driven text logic processing module (TLA) and visual semantic processing module (VSA) perform preliminary interpretation of each cross-modal task unit, and output initial text summaries respectively. and initial information description set The outputs of the two modules are aggregated to construct an initial summary set. .

[0080] Specifically, this step aims to utilize the cross-modal literature dataset obtained in step S1000. The data is then distributed to two parallel processing modules: the Text Logic Analysis Module (TLA) and the Visual Semantic Analysis Module (VSA), each of which independently generates a preliminary interpretation of the local cross-modal information, constructing an initial summary set. .

[0081] Further, step S2000 includes:

[0082] Step S2100, based on the association mapping table Traverse the collection of content units For each non-text unit, all associated text block units are queried and filtered to form a text context set. Each non-text unit and its corresponding text context set are then encapsulated into a cross-modal task unit, thus constructing a cross-modal task unit set. .

[0083] Specifically, this step aims to aggregate the relevant textual context for each non-textual object based on the established cross-modal associations. This involves using a structured cross-modal literature dataset. The collection of content units in and associated mapping table This is transformed into a set of cross-modal task units centered around non-textual objects. This provides a local and complete information input environment for the subsequent two parallel processing modules. Represents the text-visual task unit, encapsulating the first j A visual object and all its associated text descriptions; Represents a text-formula task unit, encapsulating the first... A formula object and its associated text context.

[0084] In the specific implementation process, this step involves executing a task dispatch function to associate the mapping table. Based on this, it forms a set of content units. Each non-text unit in the process constructs its corresponding task unit. The core of this process is the context aggregation operation, which, for any given visual object unit... By querying the associated mapping table From all text block units Select text blocks that meet preset conditions. A subset of that, forming its associated text set. The filtering criteria are: in the association mapping table... M There exists at least one triplet entry in the text, and the text index of that triplet entry is... i And non-text indexes are j Based on this filtering condition, a semantic association query operation is performed to extract all visual object units that are related to the given visual object. There are text contents with pre-existing relationships.

[0085] After completing the visual object unit Association with its text context set After binding, the two are encapsulated into ordered tuples to form a complete text-visual task unit. Similarly, for any given formula object unit... Using the same logical steps, corresponding text-formula task units are formed. .in, Indicates the first Formula object unit The associated set of text contexts.

[0086] Step S2200: Drive the text logic processing module TLA to perform deep semantic understanding and information compression on the text context set associated with each non-text unit, and generate an initial text summary representing the context set. .

[0087] Specifically, this step aims to drive the text logic processing module TLA to perform deep semantic understanding and information compression on the set of text contexts associated with non-text units constructed in step S2100, generating a preliminary summary of the core content of the text context set, i.e., an initial text summary set. .in, and These represent the text logic processing module TLA targeting the first... j visual object unit and the Formula object unit The associated set of text contexts, at the initial time step time= 0. The initial text summary generated; Specifically refers to the text logic processing module (TLA); Indicates the initial time step time= 0.

[0088] In its implementation, the text logic processing module TLA is a text generator based on a generative large language model, whose generation behavior is defined by pre-trained model parameters. This text generator receives all string content from the text context set as input, utilizes an encoder-decoder architecture composed of multiple self-attention mechanisms, and combines decoding strategies, such as bundle search, to perform deep semantic understanding and information integration, outputting a coherent and generalized initial text summary.

[0089] Step S2300: Drive the visual semantic processing module VSA to process the visual object units according to the category labels of the non-text units. pixel matrix Perform visual feature extraction and semantic modeling, as well as process formula object units. Source code content Perform syntax parsing and semantic understanding to generate corresponding initial information descriptions and construct an initial information description set. .

[0090] Specifically, this step aims to drive the Visual Semantic Processing (VSA) module, based on the visual object units selected in step S2100. or formula object unit Generate structured textual descriptions, i.e., the initial information description set. .in, and These represent the Visual Semantic Processing (VSA) module for the first... j visual object unit and the Formula object unit At the initial time step time= 0. Initial information description generated; Specifically refers to the Visual Semantic Processing (VSA) module; Indicates the initial time step time= 0.

[0091] In its implementation, the core of the Visual Semantic Processing (VSA) module is a multimodal large language model, which integrates a multimodal content generator with adaptive input modality processing capabilities. This multimodal content generator executes corresponding parsing and generation logic based on the category labels of the received non-textual information, as follows:

[0092] First, when the visual object unit During input, the multimodal content generator directly processes the visual object unit. Contained pixel matrix Through an internal visual encoder and language decoder, it performs feature extraction and semantic modeling, such as identifying chart types, extracting curve trends, locating key data points, and parsing architectural components, outputting results tailored to the specific context. j visual object unit Description of the generated initial information.

[0093] Second, when the formula object unit During input, the multimodal content generator processes the formula object unit. The included source code content It performs grammatical parsing and semantic understanding, analyzes its mathematical meaning, such as identifying variable relationships, explaining operational logic, and illustrating the main idea of ​​the formula, and generates a formula for the first... Formula object unit Description of the generated initial information.

[0094] Step S2400: For each cross-modal task unit, the initial text summary set generated by the text logic processing module TLA is processed. The initial information description set generated by the Visual Semantic Processing Module (VSA) Perform aggregation to construct an initial summary set. .

[0095] Specifically, this step aims to process the initial text summary set generated in step S2200. and the initial information description set generated in step S2300 Aggregate the data to build a unified and structured initial summary set. This provides formalized initial observations for subsequent steps.

[0096] In the specific implementation process, this step executes a state construction function. This function, for each completed cross-modal task unit in step S2100, combines its corresponding initial text summary and initial information description into an ordered tuple that serves as the initial joint state. This tuple encapsulates the state before the reinforcement learning optimization begins, i.e., the initial time step. time= 0. A preliminary understanding of the cross-modal task unit without co-optimization.

[0097] The process of constructing the initial joint state can be formally represented for different types of cross-modal task units as follows:

[0098] First, for the text-visual task unit Its initial joint state , constructed as .

[0099] Second, for the text-formula task unit Its initial joint state , constructed as .

[0100] in, Indicates in time= 0th minute j The initial joint state of each text-visual task unit is an ordered tuple used to encapsulate the initial text summary. and initial information description The core data structure; Shown time= 0th minute The initial joint state of each text-formula task unit is an ordered tuple used to encapsulate the initial text summary. and initial information description The core data structure.

[0101] Ultimately, the initial joint states generated by all cross-modal task units together constitute the initial summary set. .

[0102] S3000 initiates an iterative closed loop driven by cross-modal consistency rewards, based on the current joint state. Build joint actions According to the joint state The reward signal is calculated by the cross-modal consensus arbitrator (CMCA). Based on the reward signal A near-end policy optimization (PPO) algorithm is constructed to update the policy networks of the text logic processing module (TLA) and the visual semantic processing module (VSA), completing the state update and applying the reward signal. The convergence status is used to determine whether to terminate the closed loop.

[0103] Specifically, this step aims to construct the initial summary set in step S2400. Starting with the initial state, iterative optimization is performed within a closed-loop control flow of Multi-Agent Reinforcement Learning (MARL) driven by a cross-modal consistent reward function, outputting a semantically stable and consistent final joint state set. The reward function is defined by the feature vector set constructed in step S1200. F and the set of content units produced in step S1100 U This serves as the basis for calculations.

[0104] Further, step S3000 includes:

[0105] Step S3100: Drive the text logic processing module TLA and the visual semantic processing module VSA to respectively set the current joint state. Local observations are input into their respective policy networks to generate data in a predefined action space. Based on the probability distribution, the corresponding action is determined, and the actions determined by the two modules are combined into a joint action. .

[0106] Specifically, this step aims to define the core decision-making mechanism for Multi-Agent Reinforcement Learning (MARL). This mechanism drives the Text Logic Processing (TLA) module and the Visual Semantic Processing (VSA) module based on the current joint state. Local observations, utilizing their respective policy networks, output joint actions to improve the current summary hypothesis. .in, Indicates the first iter The joint state during round iterations, when dealing with visual tasks, specifically refers to the first... j Initial joint state of text-visual task units When handling formula tasks, it specifically refers to the first... Initial joint state of text-formula task units .

[0107] In the specific implementation process, in each iteration iter In this context, the Text Logic Processing (TLA) module and the Visual Semantic Processing (VSA) module act as independent decision-making units, each starting from the current joint state. The text logic processing module (TLA) extracts the self-generated content portion as a local observation and inputs it into its dedicated policy network. The local observation of the TLA is the current joint state. The text summary; the local observations of the Visual Semantic Processing (VSA) module represent the current joint state. The information description in the text.

[0108] The policy network is a deep neural network defined by trainable parameters, such as a policy evaluation network with an Actor-Critic architecture. After receiving local observations, the policy network outputs a policy in a predefined and discrete action space. The probability distribution is used as a basis. Random sampling is performed based on the probability distribution to determine the specific actions that each module will perform in this round, i.e., the text logic processing module TLA action selection. Visual Semantic Processing Module (VSA) Action Selection Ultimately, the two modules select actions independently. and Together, they construct an ordered tuple, which is the joint action of this iteration. .

[0109] Step S3200: Drive the cross-modal consensus arbitrator (CMCA) to determine the current joint state. Semantic alignment rewards Numerical consistency reward Rewards for conforming to logic The reward signal is obtained by comprehensively evaluating the three dimensions and weighting and summing the reward scores of each dimension. .

[0110] Specifically, this step aims to design and compute a reward signal that can quantitatively evaluate the consistency between the outputs of the Text Logic Processing Module (TLA) and the Visual Semantic Processing Module (VSA). The reward signal This is the core basis for the collaborative optimization of the two modules.

[0111] In its implementation, this step incorporates a Cross-Modal Consistency Arbitrator (CMCA). At the end of each iteration, the CMCA addresses the current joint state. The system performs a comprehensive evaluation based on three dimensions: semantics, quantification, and structure. The scores from each dimension are then weighted and summed to obtain the final reward signal. The details are as follows:

[0112] First dimension, semantic alignment reward The semantic alignment reward is used to evaluate the degree of semantic alignment between the text descriptions output by the two modules: the text summary and the information description. The arbitrator CMCA invokes a sentence embedding model to map the two text descriptions into high-dimensional semantic vectors and calculates a score based on cosine similarity as the semantic alignment reward. .

[0113] The second dimension is the reward for numerical consistency. The numerical consistency reward is used to verify whether the specific quantitative claims in the text description match the original non-text data. The arbitrator CMCA's processing logic adaptively adjusts based on the type of the non-text unit.

[0114] For visual objects, the reward is represented by data trend verification. The arbitrator CMCA is derived from the visual object unit. Extract quantified visual trend vectors For example, the slope of a series of key points on a curve graph. Simultaneously, the corresponding trend declaration vector is parsed from the text description of the current state. Rewards for consistency of reward values It is inversely proportional to the norm error between the two vectors.

[0115] For formula objects, the reward manifests as a numerical substitution verification. The arbitrator CMCA calls a numerical parsing function to analyze the text description of the current state to determine if a specific calculation statement of the formula exists. For example, it might parse the variable assignment pair (x=2, y=5) from the text "when x=2, y=5". If it does, a verification function is called to substitute the parsed numerical values ​​into the formula object cell. Perform verification and output a numerical consistency reward based on the Boolean value of the verification result. .

[0116] The third dimension is logical consistency and reward. This logic aligns with the reward system used to verify whether the structure, flow, or logical relationships in the text description are consistent with the original non-text data.

[0117] For visual objects, the reward is represented by topological conformity verification. The arbitrator CMCA will process the visual object units. The topological information is represented as a baseline graph structure, where nodes represent components and edges represent relationships. Simultaneously, a structure parsing function is invoked to extract the steps and component relationships from the text description of the current state and construct a text-derived graph structure. The reward value is calculated based on the graph edit distance between the two graph structures.

[0118] For formula objects, this reward is manifested as a verification of mathematical syntax structure conformity. The arbitrator CMCA will unite the formula object. The abstract syntax tree is parsed and used as the base logic tree. Simultaneously, a structure parsing function is invoked to extract declarations of the mathematical structure of the formula from the text description of the current state, such as identifying operation types, operands, and operation levels, and constructing a text-derived explicit logic tree. The reward value is calculated based on the degree of matching between the two logic trees in key nodes and hierarchical relationships.

[0119] Finally, the reward signal The sum is obtained by weighting the above sub-rewards using preset weights.

[0120] Step S3300, based on reward signal The advantage function is calculated, and the objective function of the proximal policy optimization (PPO) algorithm is constructed by combining the action probability ratio. The stochastic gradient ascent algorithm is then used to apply the parameters of the text logic processing module (TLA) along the gradient direction of the objective function. The parameter set of the policy network of the Visual Semantic Processing Module (VSA). Perform synchronized updates.

[0121] Specifically, this step aims to utilize the reward signal obtained in step S3200. As objective feedback, the parameters of the policy networks of the text logic processing module (TLA) and the visual semantic processing module (VSA) are iteratively updated, and the updated parameter set of the TLA policy network is output. The parameter set of the Visual Semantic Processing Module (VSA) policy network .

[0122] In the specific implementation process, this step employs a stable policy gradient algorithm, namely Proximal Policy Optimization (PPO), to update the network parameters of the text logic processing module (TLA) and the visual semantic processing module (VSA). The specific update process mainly includes the following computational steps:

[0123] First, the dominance function is calculated. Based on the collected empirical sequences, the dominance function is calculated in the state... Next action The advantage function. Wherein, the empirical sequence is composed of the first... iter The combined state of the wheels Joint actions and reward signals The advantage function is used to evaluate the selected action. Compared to the average action value in this state, a positive advantage value means that the current action is better than the average strategy and should be encouraged; conversely, it should be suppressed.

[0124] Second, the objective function is constructed and optimized. Based on the action probability ratio and the aforementioned advantage function, a unique objective function for the Proximal Policy Optimization (PPO) algorithm is constructed. This objective function is obtained by calculating the expected value of a key indicator on a batch of experience sequences. The action probability ratio is a key indicator that measures the change in the probability of selecting the same action in the same state before and after the policy network update. It is composed of the ratio of the probability output of the new policy network executing the current optimization to the probability output of the old policy network that performs sampling to collect experience data. This key indicator is determined by selecting one of two candidate values ​​through a minimum value function, thus achieving the conservative principle of policy update.

[0125] The first candidate value is an unconstrained policy optimization objective. This term directly multiplies the action probability ratio by the advantage function, adjusting the policy proportionally based on the sign of the advantage function. The second candidate value is a constrained policy optimization objective. This term introduces a pruning mechanism, forcibly constraining the action probability ratio to a set of hyperparameters. Defined narrow trust area This constrains the step size of a single network parameter update, preventing policy performance from collapsing due to excessively large updates and ensuring the stability of the learning process. Subsequently, the ratio of the pruned action probabilities is multiplied by the advantage function.

[0126] Third, parameter gradient update. The Stochastic Gradient Ascent (SGA) algorithm is used, based on the objective function... The gradient direction, for a unified set of parameters of the policy network. The optimization and update process is as follows: The specific formula is as follows:

[0127] ;

[0128] in, Indicates the first iter A unified set of parameters after rounds of iterative updates; Indicates the first iter A unified set of parameters before each round of iterative updates; This represents the unified set of parameters for all policy networks to be optimized, including The parameter set of the TLA policy network representing the text logic processing module and This represents the parameter set of the Visual Semantic Processing (VSA) policy network. This represents the learning rate during the parameter update process, used to control the step size of a single parameter update; Describe the objective function For a unified parameter set In the iter Unified parameter set before each iteration update The gradient calculated at that point.

[0129] Ultimately, a unified parameter set was achieved. From its in the iter Initial state of round iteration Evolves into the optimized state This update will be applied synchronously to its subset, namely the parameter set of the Text Logic Processing Module (TLA) policy network. The parameter set of the Visual Semantic Processing Module (VSA) policy network The optimized state mentioned above. It will be enabled when the process returns to the next round in step S3100 to drive the parameters of the policy network to generate better actions, forming a feedback control closed loop.

[0130] Step S3400, based on the combined action Regarding the current joint state Perform state updates to generate the joint state for the next iteration. Monitor reward signals In continuous iterations, if the absolute value of the relative rate of change is consistently lower than a preset convergence threshold, the iteration loop is terminated and the final joint state set is output. Otherwise, the updated joint state will be fed back to the decision-making step to start the next iteration.

[0131] Specifically, this step aims to address the combined action output by step S3100. Regarding the current joint state Perform a state update to generate the joint state for the next iteration. Simultaneously, by continuously monitoring changes in the reward signal, it determines whether the optimization process has reached the convergence condition, thereby deciding whether to continue iterating or terminate the process and output the final joint state set. .

[0132] In the specific implementation process, this step includes two parallel processing logics: state transition and convergence judgment, as detailed below:

[0133] First, state transition. Based on the... iter Joint actions of round iteration The specific instructions update the current union state. Generate the next round of joint state .in, and They represent the first iter At the start of each iteration, the current joint state is formed. Text summary and information description.

[0134] If any module's action is "modify" or "rewrite", then that module will invoke its internal generator based on the current union state. Regenerate the corresponding description section to obtain the updated text summary. and information description As the next round of joint status The corresponding content. Among them, and They represent the first time. iter Temporary content newly generated by modules that take "correction" or "rewrite" actions during rounds of iteration.

[0135] If any module's action is "hold", then the current union state is... The corresponding text summary and information description remain unchanged.

[0136] In generating the next round of joint states Then, the iteration round index is incremented, and the control flow returns to step S3100 to start a new round of decision-making.

[0137] Second, convergence judgment. To avoid infinite iteration and ensure stable output, the reward signal is continuously monitored. The change in the reward signal. If the absolute value of the relative rate of change in several consecutive iterations is less than a preset convergence threshold, the iterative process in the MARL closed-loop control flow of the multi-agent reinforcement learning system is determined to have converged, and the loop is terminated. At this point, the joint state of the current iteration is marked as the final joint state, i.e., the final joint state set. .

[0138] S4000, for the final joint state set Deconstruction is performed to separate the final text summary set. and the final information description set To associate the mapping table M Guided by a structured approach, a sequence-to-sequence language model is used to fuse and reconstruct the two separated sets, generating the final summary. .

[0139] Specifically, this step aims to process the final joint state set output by step S3400. Through a context-aware fusion generation process, a complete and coherent final summary is synthesized. This allows for the integration of fragmented information into a coherent and logically clear text that conforms to human reading habits.

[0140] Further, step S4000 includes:

[0141] Step S4100, for the final joint state set The data is reconstructed by extracting and separating the encapsulated text summaries and information descriptions to construct the final text summary set. and the final information description set .

[0142] Specifically, this step aims to refine the final joint state set. Data reconstruction is performed by extracting and separating the text-based summaries from the non-text-based descriptions, and outputting a final aggregated set of text summaries. and the final information description set .

[0143] The final joint state set It is the final joint state set of all text-visual task units. The final joint state set of all text-formula task units The union of .

[0144] In the specific implementation, a fragment extraction function is used to traverse the final joint state set one by one. Each member in the process, i.e., each final joint state .in, xThis represents a general index, which can correspond to the index of a visual object. j or index of formula object ; and These represent the general indexes respectively. x The final union state referred to Text summaries and information descriptions within.

[0145] For each joint state The fragment extraction function performs an unpacking operation, separating each final joint state. Text summary Extract it and add it as an element to the final text summary set. Similarly, for each final joint state... Information description Extract it and add it as an element to the final information description set. middle.

[0146] Step S4200, using the associated mapping table M As a structured context guide, it drives the sequence-to-sequence language model to process the final text summary set. and the final information description set Perform a natural language generation task to generate the final summary. .

[0147] Specifically, this step aims to extract the final text summary set output by step S4100. and the final information description set The association mapping table constructed in step S1300 M As a structured guide, a generative model is used for intelligent fusion and reconstruction to output a final abstract that is logically clear, structurally sound, and reflects the core innovations of the original text. .

[0148] In the specific implementation process, this step enables a summary fusion module, the core of which is a large-scale sequence-to-sequence (Seq2Seq) language model that has been fine-tuned by text rewriting and document generation tasks. Its function is defined by the conditional fusion function.

[0149] The conditional fusion function takes all optimized summary fragments as input, i.e., the final text summary set. and the final information description set At the same time, using cross-modal association mapping tables M This serves as a structured context condition, thereby obtaining prior knowledge of the text structure. Specifically, the structured context refers to the association mapping table... MThe information provided goes beyond the semantics of a single sentence, including the spatial proximity between summary segments and the display of citation relationships.

[0150] Based on this structured context, the fusion function performs a series of natural language generation tasks, including content planning, sentence rearrangement, cohesion and coherence generation, and citation reconstruction, ultimately producing a coherent and logically sound summary. Content planning determines the order in which topics or key points are presented; sentence rearrangement organizes semantically related summary segments into adjacent sections; cohesion and coherence generation adds necessary transition words and logical connectors between summary segments; and citation reconstruction generates citations that conform to the logic of the original text, such as "as..." Figure 1 As shown.

[0151] Example 2

[0152] This embodiment, based on Embodiment 1, provides an LLM output stability control system based on reinforcement learning, such as... Figure 2 As shown, it includes a document modality segmentation module, a dual-module parallel interpretation module, a cross-modal reward arbitration module, and a joint state deconstruction module;

[0153] The document modality segmentation module is used for processing unstructured document data. Perform page layout analysis to segment and extract multimodal content regions, and convert the content regions into sets of content units. U For the set of content units U Each content unit in the text is vectorized to generate a set of feature vectors. And construct cross-modal association mappings between the content units to build an association mapping table. M The document data Convert to a collection containing content units U eigenvector set F and associated mapping table M Cross-modal literature dataset ;

[0154] The dual-module parallel interpretation module is used to interpret data based on the association mapping table. Each non-text unit is grouped into a text context set to construct a cross-modal task unit set. The parallel-driven text logic processing module (TLA) and visual semantic processing module (VSA) perform preliminary interpretation of each cross-modal task unit, and output initial text summaries respectively. and initial information description set The outputs of the two modules are aggregated to construct an initial summary set. ;

[0155] The cross-modal reward arbitration module is used to initiate an iterative closed loop driven by cross-modal consistency rewards, based on the current joint state. Build joint actions According to the joint state The reward signal is calculated by the cross-modal consensus arbitrator (CMCA). Based on the reward signal A near-end policy optimization (PPO) algorithm is constructed to update the policy networks of the text logic processing module (TLA) and the visual semantic processing module (VSA), completing the state update and applying the reward signal. The convergence status is used to determine whether to terminate the closed loop;

[0156] The joint state deconstruction module is used to decompose the final joint state set. Deconstruction is performed to separate the final text summary set. and the final information description set To associate the mapping table M Guided by a structured approach, a sequence-to-sequence language model is used to fuse and reconstruct the two separated sets, generating the final summary. .

[0157] The parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0158] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A reinforcement learning-based LLM output stability control method, characterized in that, include: Page layout analysis is performed on unstructured literature data to segment and extract multimodal content regions. The content regions are converted into a set of content units. Each content unit in the set of content units is vectorized to generate a set of feature vectors. Cross-modal association mapping between the content units is constructed to build an association mapping table. The literature data is then converted into a cross-modal literature dataset containing a set of content units, a set of feature vectors, and an association mapping table. Based on the association mapping table, each non-text unit is formed into a text context set to construct a cross-modal task unit set. The text logic processing module and the visual semantic processing module are driven in parallel to perform preliminary interpretation of each cross-modal task unit, and output initial text summaries and initial information description sets respectively. The outputs of the two modules are aggregated to construct an initial summary set. Initiate an iterative closed loop driven by cross-modal consistency rewards, construct joint actions based on the current joint state, calculate reward signals by a cross-modal consistency arbitrator based on the joint state, construct a proximal policy optimization algorithm based on the reward signals to update the policy networks of the text logic processing module and the visual semantic processing module, complete the state update, and determine whether to terminate the closed loop based on the convergence of the reward signals. The final joint state set is deconstructed to separate the final text summary set and the final information description set. Using the association mapping table as a structured guide, the sequence-to-sequence language model is driven to fuse and reconstruct the two separated sets to generate the final summary.

2. The LLM output stability control method based on reinforcement learning according to claim 1, characterized in that, The transformation steps of the cross-modal literature dataset include: Page layout analysis and semantic segmentation are performed on unstructured literature data to identify and extract text, image and formula content areas with category labels and bounding boxes, and the extracted content areas are converted into a set of structured content units. Based on the category label of each content unit in the content unit set, the corresponding text or visual encoder based on the Transformer architecture is called to encode the content unit into a corresponding feature vector, forming a feature vector set containing text, visual and formula semantics. For the combination pairs of text block units and non-text units in the content unit set, a comprehensive association scoring algorithm is executed to calculate the explicit reference relationship score and the spatial proximity score respectively. The two scores are weighted to generate an association confidence score. The combination pairs with the association confidence score exceeding the preset confidence threshold are selected, and an association mapping table is constructed. By executing all the above processes, a cross-modal literature dataset is obtained.

3. The LLM output stability control method based on reinforcement learning according to claim 1, characterized in that, The method for constructing the initial summary set includes: Based on the association mapping table, traverse each non-text unit in the content unit set, query and filter all associated text block units for the non-text unit to form a text context set, encapsulate each non-text unit and its corresponding text context set into a cross-modal task unit, and construct a cross-modal task unit set. The driving text logic processing module performs deep semantic understanding and information compression on the text context set associated with each non-text unit, and generates an initial text summary representing the context set. The visual semantic processing module is driven to perform visual feature extraction and semantic modeling on the pixel matrix of the visual object unit according to the category label of the non-text unit, and to perform syntax parsing and semantic understanding on the source code content of the formula object unit, generating the corresponding initial information description and constructing the initial information description set. For each cross-modal task unit, the initial text summary set generated by the text logic processing module and the initial information description set generated by the visual semantic processing module are aggregated to construct an initial summary set.

4. The LLM output stability control method based on reinforcement learning according to claim 3, characterized in that, The filtering method for the text context set includes having at least one triplet entry in the association mapping table, the triplet entry having a corresponding text index and a corresponding non-text index, and completing a semantic association query operation based on the filtering method to extract all text content that has a predetermined association with a given visual object unit.

5. The LLM output stability control method based on reinforcement learning according to claim 1, characterized in that, The update steps for the policy networks of the text logic processing module and the visual semantic processing module include: The text logic processing module and the visual semantic processing module are driven to input the local observations of the current joint state into their respective policy networks, generate probability distributions in the predefined action space, determine the corresponding actions based on the probability distributions, and construct a joint action by the actions determined by the two modules. The cross-modal consistency arbitrator is driven to comprehensively evaluate the current joint state from three dimensions: semantic alignment reward, numerical consistency reward, and logical compliance reward. The reward signal is obtained by weighted summation of the reward scores of each dimension. The advantage function is calculated based on the reward signal, and the objective function of the near-end policy optimization algorithm is constructed by combining the action probability ratio. The stochastic gradient ascent algorithm is used to synchronously update the parameter set of the text logic processing module and the parameter set of the policy network of the visual semantic processing module along the gradient direction of the objective function.

6. The LLM output stability control method based on reinforcement learning according to claim 5, characterized in that, The semantic alignment reward is used to evaluate the degree of alignment between the text summary and the information description at the semantic level; the numerical consistency reward is used to verify whether the specific quantitative statements in the text description are consistent with the original non-text data; and the logical consistency reward is used to verify whether the structure, process, or logical relationship in the text description is consistent with the original non-text data.

7. The LLM output stability control method based on reinforcement learning according to claim 1, characterized in that, The method for determining the termination of the closed loop includes: updating the current joint state based on the joint action to generate the joint state for the next iteration; monitoring the relative rate of change of the reward signal in the continuous iteration; if the absolute value of the relative rate of change is continuously lower than a preset convergence threshold, terminating the iteration loop and outputting the final set of joint states; otherwise, feeding back the updated joint state to the decision-making step to start the next iteration.

8. The LLM output stability control method based on reinforcement learning according to claim 1, characterized in that, The method for generating the final summary includes: The final joint state set is reconstructed by extracting and separating the encapsulated text summary and information description to construct the final text summary set and the final information description set. Using an association mapping table as a structured context guide, a sequence-to-sequence language model is driven to perform natural language generation tasks on the final text summary set and the final information description set to generate the final summary.

9. A reinforcement learning-based LLM output stability control system, used to implement the reinforcement learning-based LLM output stability control method according to any one of claims 1-8, characterized in that, The system includes a document modality segmentation module, a dual-module parallel interpretation module, a cross-modal reward arbitration module, and a joint state deconstruction module; The document modality segmentation module is used to perform page layout analysis on unstructured document data to segment and extract multimodal content regions, convert the content regions into a set of content units, perform feature vectorization on each content unit in the set of content units to generate a set of feature vectors, and construct a cross-modal association mapping between the content units to build an association mapping table, thereby converting the document data into a cross-modal document dataset containing a set of content units, a set of feature vectors, and an association mapping table. The dual-module parallel interpretation module is used to form a text context set of each non-text unit according to the association mapping table to construct a cross-modal task unit set, and to drive the text logic processing module and the visual semantic processing module in parallel to perform preliminary interpretation of each cross-modal task unit, respectively outputting an initial text summary and an initial information description set. The outputs of the two modules are aggregated to construct an initial summary set. The cross-modal reward arbitration module is used to initiate an iterative closed loop driven by cross-modal consistency rewards, construct joint actions based on the current joint state, calculate reward signals by the cross-modal consistency arbitrator based on the joint state, construct a proximal policy optimization algorithm based on the reward signals to update the policy networks of the text logic processing module and the visual semantic processing module, complete the state update, and determine whether to terminate the closed loop based on the convergence of the reward signals. The joint state deconstruction module is used to deconstruct the final joint state set to separate the final text summary set and the final information description set. Using the association mapping table as a structured guide, it drives the sequence-to-sequence language model to fuse and reconstruct the two separated sets to generate the final summary.

Images