LLM output stability control method and system based on reinforcement learning

Abstract

The invention provides an LLM output stability control method and system based on reinforcement learning, and the method comprises the steps: carrying out the multi-modal structured analysis of literature data, extracting a content unit, and constructing a feature vector set and an association mapping table; the text logic processing module and the visual semantic processing module are driven in parallel to interpret the cross-modal task unit, and an initial abstract set is constructed; in a cross-modal consistency reward-driven iterative closed loop, constructing a joint action according to a current joint state, calculating a reward signal by a cross-modal consistency arbiter, and constructing a near-end strategy optimization algorithm to update a strategy network until the reward signal converges; and based on the association mapping table, the sequence-to-sequence language model is driven to reconstruct the final text abstract and the information description set of the final joint state set, and a final abstract is generated. Through structured analysis and multi-mode cooperative control, the consistency and stability of abstract contents are ensured, and the method is suitable for the field of natural language processing and reinforcement learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing and reinforcement learning, in particular to a LLM output stability control method and system based on reinforcement learning. BACKGROUND

[0002] With the wide application of large-scale language models (LLM) in the field of intelligent analysis and knowledge discovery of scientific and technological literature, how to effectively control a single end-to-end language model to generate a summary that is both faithful to the original content and logically consistent across modal information when dealing with complex literature containing text, charts, formulas and other multi-modal elements has become a key technical problem in the process of moving from single-point content generation to system-level stability control of large language models.

[0003] Chinese patent application CN119884355B provides a LLM text processing method, which includes: obtaining input question information; inputting the input question information into a word segmentation database to output question words; inputting the question words into a preset word database to output synonymous words, mutually exclusive words and abbreviated words, and defining the question words, synonymous words, mutually exclusive words and abbreviated words as inquiry words; outputting the final question information according to the inquiry words; finding out the reference answer information and the corresponding path through the preliminary examination method according to the final question information; inputting the reference answer information and the path into a priority database to match the priority; determining the preliminary examination answer information from the matched priority; finding out the refined examination answer information through the refined examination method according to the preliminary examination answer information; taking the refined examination answer information as the final answer text and uploading and displaying it.

[0004] However, the current technology still faces many challenges. The application bottleneck of current large-scale language models is concentrated on the internal consistency of processing cross-modal information. When dealing with key scientific and technological literature such as clinical drug trial reports containing complex charts, the model generally lacks a built-in mechanism for cross-validation of text conclusions and chart data. Not only does this lead to inconsistencies between the text conclusions output by the model and the quantitative data revealed by the charts, but it also often leads to deep semantic contradictions. For example, the text summary may assert that the drug has a significant effect, while the error bar data in the chart shows that there is no statistically significant difference. This bias, which cannot be self-corrected by the model, makes the output summary lose its scientific reliability, and in turn leads to a series of serious misjudgments. In basic research, this can lead to errors in subsequent research directions, resulting in a huge waste of scientific research resources; in key fields such as biomedicine, misleading summary conclusions can directly interfere with clinical treatment plans, posing a potential threat to patients' health and safety. SUMMARY

[0005] To achieve the above object, the application provides a LLM output stability control method based on reinforcement learning, and the specific technical solutions are as follows: Performing page layout analysis on unstructured literature data to segment and extract multi-modal content regions, converting the content regions into a content unit set, performing feature vectorization on each content unit in the content unit set to generate a feature vector set, and constructing a cross-modal association mapping between the content units to construct an association mapping table, and converting the literature data into a cross-modal literature data set containing the content unit set, the feature vector set, and the association mapping table; According to the association mapping table, each non-text unit forms a text context set to construct a cross-modal task unit set, and a text logic processing module and a visual semantic processing module are driven in parallel to perform preliminary interpretation on each cross-modal task unit, and an initial text summary and an initial information description set are output respectively, and the outputs of the two modules are aggregated to construct an initial summary set; An iterative closed loop driven by cross-modal consistency reward is started, a joint action is constructed according to the current joint state, a reward signal is calculated by a cross-modal consistency arbitrator according to the joint state, a proximal policy optimization algorithm is constructed based on the reward signal to update the policy network of the text logic processing module and the visual semantic processing module, the state is updated, and it is judged whether to terminate the closed loop according to the convergence of the reward signal; The final joint state set is deconstructed to separate the final text summary set and the final information description set, and a sequence-to-sequence language model is driven to reconstruct the two separated sets based on the structured guidance of the association mapping table to generate a final summary.

[0006] Further, the conversion step of the cross-modal literature data set comprises: Performing page layout analysis and semantic segmentation on unstructured literature data, identifying and extracting text, image and formula content regions with category labels and boundary boxes, and converting the extracted content regions into a structured content unit set; According to the category labels of each content unit in the content unit set, a corresponding text or visual encoder based on the Transformer architecture is called to encode the content unit into a corresponding feature vector to form a feature vector set containing text, visual and formula semantics; For the combination of text block units and non-text units in the content unit set, an integrated association scoring algorithm is performed to calculate the explicit reference relationship score and the spatial proximity score respectively, the two scores are weighted to generate an association confidence score, and the combination pairs with the association confidence score exceeding the preset confidence threshold are selected to construct an association mapping table; By performing the above-mentioned entire process, a cross-modal literature data set is obtained.

[0007] Further, the construction method of the initial summary set comprises: Based on the association mapping table, traversing each non-text unit in the content unit set, querying and screening all associated text block units for the non-text unit to form a text context set, encapsulating the non-text unit and its corresponding text context set as a cross-modal task unit, and constructing a cross-modal task unit set; Driving the text logic processing module to perform deep semantic understanding and information compression on the text context set associated with each non-text unit, and generating an initial text summary representing the context set; Driving the visual semantic processing module to perform visual feature extraction and semantic modeling on the pixel matrix of the visual object unit according to the category label of the non-text unit, and performing syntax analysis and semantic understanding on the source code content of the formula object unit, generating corresponding initial information description, and constructing an initial information description set; For each cross-modal task unit, aggregate the initial text summary set generated by the text logic processing module and the initial information description set generated by the visual semantic processing module to construct an initial summary set.

[0008] Further, the screening method of the text context set comprises that there is at least one triple entry in the association mapping table, and the triple entry has a corresponding text index and a corresponding non-text index. Based on the screening method, a semantic association query operation is performed to extract all text content that has a predetermined association with the given visual object unit.

[0009] Further, the update step of the strategy network of the text logic processing module and the visual semantic processing module two modules comprises: Driving the text logic processing module and the visual semantic processing module, respectively inputting the local observation value of the current joint state into the respective strategy network, generating a probability distribution on the predefined action space, determining the corresponding action according to the probability distribution, and constructing the joint action from the actions determined by the two modules; Driving the cross-modal consistency arbitrator to comprehensively evaluate the current joint state from three dimensions of semantic alignment reward, numerical consistency reward and logical compliance reward, and weighting and summing the reward scores of the dimensions to obtain a reward signal; Based on the reward signal, calculate the advantage function, combine the action probability ratio to construct the objective function of the proximal policy optimization algorithm, and use the stochastic gradient ascent algorithm along the gradient direction of the objective function to update the parameter set of the text logic processing module and the parameter set of the strategy network of the visual semantic processing module.

[0010] Furthermore, the semantic alignment reward is used to evaluate the degree of alignment between the text summary and the information description at the semantic level; the numerical consistency reward is used to verify whether the specific quantitative statements in the text description are consistent with the original non-text data; and the logical consistency reward is used to verify whether the structure, process or logical relationship in the text description is consistent with the original non-text data.

[0011] Furthermore, the method for determining the termination of the closed loop includes: updating the current joint state based on the joint action to generate the joint state for the next round of iteration, monitoring the relative change rate of the reward signal in continuous iterations, and terminating the iterative loop to output the final joint state set if the absolute value of the relative change rate is continuously lower than a preset convergence threshold; otherwise, feeding back the updated joint state to the decision step to start the next round of iteration.

[0012] Furthermore, the method for generating the final summary includes: Reconstruct the data of the final joint state set, extract and separate the encapsulated text summary and information description, and construct the final text summary set and final information description set; Using the association mapping table as a structured context guide, the sequence-to-sequence language model is driven to perform the natural language generation task on the final text summary set and the final information description set to generate the final summary.

[0013] A reinforcement learning-based LLM output stability control system, which is used to implement the above-mentioned reinforcement learning-based LLM output stability control method, including a document modality segmentation module, a dual-module parallel interpretation module, a cross-modality reward arbitration module, and a joint state deconstruction module; The document modality segmentation module is configured to perform page layout analysis on unstructured document data to segment and extract multimodal content regions, convert the content regions into content unit sets, perform feature vectorization on each content unit in the content unit set to generate a feature vector set, construct a cross-modal association mapping between the content units to construct an association mapping table, and convert the document data into a cross-modal document dataset comprising a content unit set, a feature vector set, and an association mapping table; The dual-module parallel interpretation module is used to form a text context set for each non-text unit according to the association mapping table to construct a cross-modal task unit set, and drive the text logic processing module and the visual semantic processing module in parallel to perform preliminary interpretation on each cross-modal task unit, output an initial text summary and an initial information description set respectively, and aggregate the outputs of the two modules to construct an initial summary set; The cross-modal reward arbitration module is configured to start an iterative closed loop driven by a cross-modal consistency reward, construct a joint action according to a current joint state, calculate a reward signal by a cross-modal consistency arbitrator according to the joint state, construct a proximal policy optimization algorithm based on the reward signal to update a policy network of the two modules of the text logical processing module and the visual semantic processing module, complete state updating, and determine whether to terminate the closed loop according to a convergence condition of the reward signal. The joint state deconstruction module is configured to deconstruct a final joint state set to separate a final text summary set and a final information description set, use a correlation mapping table as a structured guide, drive a sequence-to-sequence language model to fuse and reconstruct the two separated sets, and generate a final summary.

[0014] Compared with the prior art, the present application has the following beneficial effects: The present application solves the problems of fuzzy and lost cross-modal semantic association caused by traditional models due to processing of flat data stream and lack of structured prior knowledge by constructing a correlation mapping table fusing explicit text reference relationship and physical space proximity, converting the implicit and layout-dependent graphic-text correspondence in the document into explicit structured correlation that can be directly parsed by a machine.

[0015] The present application solves the problems of modal feature interference and semantic focus drift caused by early information fusion of traditional single models by constructing a cross-modal task unit and driving two independent modules to perform parallel decoupled interpretation, decomposing the complex mixed modal understanding task into two highly focused single modal analysis subtasks, and providing clear and unbiased initial input for subsequent objective cross-modal consistency arbitration.

[0016] The present application solves the pseudo-alignment problem caused by the prior art relying only on a single similarity index by designing a multi-dimensional consistency reward mechanism fusing macro semantic alignment, micro numerical verification and deep logical compliance, converting the collaborative goal between the two independent modules from single and fuzzy semantic similarity into a multi-objective optimization problem containing hard constraints.

[0017] The present application solves the problems of logical flow breakage and loss of core graphic-text reference relationship of the final summary caused by traditional fusion methods due to lack of macro structure information and only local text optimization by using the correlation mapping table as the prior knowledge of the chapter structure to guide the generation process of the final summary from unordered fragment splicing to a constrained fusion reconstruction task. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 is the principle flowchart of the LLM output stability control method based on reinforcement learning of the present application; Figure 2 is the functional module diagram of the LLM output stability control system based on reinforcement learning of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0021] Embodiment 1 Please refer to Figure 1 , the present embodiment provides a LLM output stability control method based on reinforcement learning, which comprises: S1000, for unstructured literature data Performing page layout analysis to segment and extract multi-modal content regions, converting the content regions into a content unit set U , the content unit set U Each content unit is vectorized to generate a feature vector set , and a cross-modal association mapping between the content units is constructed to construct an association mapping table M , the literature data Is converted into a cross-modal literature data set containing a content unit set U , a feature vector set F And an association mapping table M .

[0022] Specifically, this step aims to convert the literature data stored in PDF format , from unstructured format readable by humans but difficult for machines to directly parse, into structured cross-modal literature data set ​By accurately segmenting and cross-modality associating the multi-modal elements such as texts, icons, formulas, etc. contained in the literature, the data is unified and machine-readable, providing a reliable data foundation for subsequent stable and consistent abstract generation.

[0023] Further, the step S1000 comprises: The step S1100, the unstructured literature data is analyzed and segmented semantically, and the text, image and formula content area with category label and boundary box are recognized and extracted, and the extracted content area is converted into a structured content unit set .

[0024] Specifically, the present step aims to decode and reconstruct the input single unstructured literature data into a set of content units composed of discrete data objects , realizing the modal separation of the original document, i.e. stripping the text, visual and formula content from the mixed layout page, and assigning each stripped object with a clear category label and physical location attribute, providing clear boundary and non-interfering data input for the subsequent steps.

[0025] In the specific implementation process, the present step is automatically executed by a document parsing engine deployed on the server side, which converts the input unstructured literature data stream into a structured content unit set by executing a comprehensive document parsing function.

[0026] First, each page of the input literature data is rasterized, which is uniformly converted from vector or mixed format to pixel matrix representation for visual model processing. Subsequently, the page layout analysis model based on deep neural network built in the document parsing engine, such as the Transformer-based LayoutLM variant architecture, analyzes the pixel matrix. The page layout analysis model identifies and locates each content area with independent function within each page by executing pattern recognition algorithms such as semantic segmentation or object detection, and outputs a page set containing multiple area descriptions. Each area is defined by its category label and boundary box coordinates . Among them, represents the category label of the k th area identified by the page layout analysis model, whose value range is limited to text TEXT, image IMAGE or formula FORMULA; represents the boundary box of the k th area on the page, which is usually defined by a set of coordinate values to describe the physical location and size of the area; kRepresents the index of an independent content area within a page. It is used to traverse and uniquely identify the content areas identified by the page layout analysis model. Its value range is 1 to the total number of areas identified on the current page.

[0027] Then, according to the category label Perform subsequent differentiated content extraction for each content area, specifically: First, for text areas classified as TEXT, call the Optical Character Recognition (OCR) algorithm to extract its digitized string content. , the string content With its bounding box Together they form a text block unit .in, Indicates the i The string content of a text block unit is a digitized text sequence extracted by the OCR algorithm; i Indicates the sequence index of the text block unit, and its value range is from 1 to the total number of text blocks; t A prefix identifier indicating the modality type, specifically the text Text, used to distinguish different modalities in variable naming; Indicates the i The bounding box of a text block unit is used to describe the spatial position of the text block unit on the page; Indicates the i A text block unit is a structured data object that encapsulates all the information of a single text block, including the string content of the text block. and its bounding box . Second, for the visual area of ​​category IMAGE, obtain its standardized pixel matrix through the standard image decoding library , the pixel matrix With its bounding box Together they form a visual object unit .in, Indicates the j The pixel data of each visual object unit is a standardized pixel matrix obtained after processing by the standard decoding library and can be directly processed by the computer vision model; j Indicates the sequence index of the visual object unit, and its value range is from 1 to the total number of visual objects; v A prefix identifier indicating the modality type, specifically Visual, used to distinguish different modalities in variable naming; Indicates the j The bounding box of each visual object unit is used to describe the spatial position of the visual object on the page; Indicates thej A visual object unit is a structured data object that encapsulates all the information of a single visual object, including the pixel data of the visual object. and its bounding box . Third, for formula areas with the FORMULA category, first try to parse the embedded MathML or LaTeX structured source code to obtain the source code content. If the parsing is successful, the source code content will be bounded by its Together they form a formula object unit If the parsing fails or the source code does not exist, the area is processed as a special visual object and classified into the aforementioned visual object unit. In the collection. Among them, Indicates the Source code content of a formula object unit , is a text string that describes the mathematical structure of the formula; Indicates the sequence index of the formula object unit, and the value range is 1 to the total number of successfully parsed formula objects; e A prefix identifier indicating the modal type, specifically the formula Equation, used to distinguish different modes in variable naming; Indicates the The bounding box of a formula object unit is used to describe the spatial position of the formula object on the page; Indicates the A formula object unit is a structured data object that encapsulates all the information of a single formula object, including the source code content of the formula object. and its bounding box .

[0028] Finally, by executing the above process, the document parsing function converts the entire document data into Completely mapped and converted into a structured set of content units , achieving the key transformation from unstructured data with fixed layout to structured multimodal content units, laying a solid data foundation for subsequent feature extraction and cross-modal correlation analysis. By all text block units The set of , all visual object units The set of and all formula object cells The set of Together they form.

[0029] Step S1200: Based on the content unit set The category labels of each content unit are called, and the corresponding text or visual encoder based on the Transformer architecture is called to encode the content unit into a corresponding feature vector to form a feature vector set containing text, visual and formula semantics .

[0030] Specifically, this step aims to map the discrete and heterogeneous content unit set output by step S1100 to a feature vector set composed of high-dimensional real vectors , realizing the numerical and semantic representation of different modal content.

[0031] In the specific implementation process, this step constructs and calls a set of pre-trained deep learning encoder models, and realizes it through a comprehensive feature extraction function. For each unit in the input content unit set U , a distributed representation in a high-dimensional vector space is generated according to its modal type, as follows: First, for the text block unit , a text encoder based on the Transformer architecture , such as the SciBERT model, is called. The text encoder takes the string content in the text block unit as input, and outputs a text feature vector that can capture deep context semantics through multi-layer self-attention mechanism calculation. Wherein, represents the text feature vector of the i th text block unit , which is used to numerically encode the semantic information of the text block unit .

[0032] Second, for the visual object unit , a visual encoder , such as the ViT model based on the Transformer, is called. The visual encoder takes the pixel matrix in the visual object unit as input, and outputs a visual feature vector representing its visual content by layer-by-layer extraction from low-order texture to high-order abstraction. Wherein, represents the visual feature vector of the j th visual object unit , which is used to numerically encode the visual pattern and structure of the text block unit . It is worth noting that the formula area that is considered as a visual object due to parsing failure in step S1100 is also processed at this link.

[0033] Third, for the formula object cell , using its source code content Essentially, it is a special sequence language with strict grammar. Reuse the text encoder Process it and output a formula feature vector that can represent its mathematical structure and semantics .in, Indicates the Formula object cell Formula feature vector, used to identify the formula object unit The mathematical structure and semantics of the data are numerically encoded.

[0034] Finally, by executing the above process, the feature extraction function sets the entire content unit Completely converted into a unified and computable set of eigenvectors , completing the deep feature representation of the multimodal content of the document. Among them, the feature vector set All text feature vectors The set of , all visual feature vectors The set of And all the formula eigenvectors The set of Together they form.

[0035] Step S1300: for a content unit set U For each pair of text block units and non-text units, we perform a comprehensive association scoring algorithm to calculate explicit citation relationship scores. and spatial proximity score , weight the two scores to generate an association confidence score, filter out the combination pairs whose association confidence scores exceed the preset confidence threshold, and construct an association mapping table M .

[0036] Specifically, this step aims to establish explicit semantic alignment between text content and non-text content, which is the set of discrete content units produced in step S1100. U Constructing a cross-modal quantitative association mapping table M The association mapping table M As the core data structure, it is used to characterize the correspondence and semantic dependencies between text block objects, visual object units and formula object units, and provide context constraints and association basis for consistency judgment and summary generation in subsequent collaborative steps.

[0037] In the specific implementation process, this step targets a set of discrete content units. U Any text block unit in In combination with non-text units, both perform a comprehensive association scoring algorithm to generate a quantitative association confidence score. This comprehensive association scoring algorithm is implemented by weighted summation of two core indicators, as follows: First, explicit reference relationship score . This indicator is used to assess whether there is a direct reference to non-text units in the text content . Among them, the non-text unit is a visual object unit or a formula object unit . By performing regular expression matching or named entity recognition in the string content of the text block unit , retrieve predefined keyword patterns. For example, for text-visual pairs, retrieve patterns such as "as shown in Figure j ", "see Table j ", etc.; for text-formula pairs, retrieve patterns such as "as formula ", "obtainable from equation ". If there is an explicit reference, the score is 1, otherwise it is 0.

[0038] Second, spatial proximity score . This indicator is used to assess the proximity of any text block unit U and non-text unit in the content unit set on the physical page of the document. Among them, the non-text unit is a visual object unit or a formula object unit . Based on the bounding box information of each unit obtained in step S1100, the Euclidean distance between the geometric center points of the text block unit and the non-text block unit is calculated, and this Euclidean distance is converted into a proximity score in the range of (0, 1] using a Gaussian kernel function. The closer the physical distance between the text unit and the non-text unit, the higher the score.

[0039] Based on the above explicit reference relationship score and spatial proximity score two scoring indicators, the association confidence score is calculated by weighted summation. The association confidence score is the final quantitative basis for determining whether there is a strong semantic association between a text unit and a non-text unit. This calculation process is performed separately for the two cross-modal combinations, as follows: First, for the text block unit and the visual object unit , the calculation formula of the association confidence score is as follows: ; wherein, Indicates the i The text block unit and the j The association confidence score between visual object units is a final evaluation value that integrates textual explicit references and spatial physical proximity and is used to measure the strength of the association between the two; Represents the weight hyperparameter, which is a floating point number with a value range of [0,1] and is used to balance the explicit reference relationship score Spatial proximity score The weight of Indicates the i The text block unit and the j The explicit reference relationship score between visual object units is a binary value, i.e. 0 or 1, which is used to represent the text block unit. Does the content contain visual units? Direct textual reference, such as "See Figure j ”; Indicates the i The text block unit and the j The spatial proximity score between visual object units is a real value in the interval (0, 1], which is used to quantify the physical proximity between the two on the document layout.

[0040] Second, for text block units With formula object cell , and its associated confidence score The calculation is as follows: ; in, Indicates the i The text block unit and the The association confidence score between formula object units is a final evaluation value that integrates textual explicit references and spatial physical proximity and is used to measure the strength of the association between the two. Indicates the i The text block unit and the The explicit reference relationship score between formula object units is a binary value, i.e. 0 or 1, which is used to represent the text block unit. Does the content contain formula object cells? Direct textual reference, such as " shown”; Indicates the i Text block units and The spatial proximity score between two formula object units is a real value in the interval (0, 1], which is used to quantify the physical proximity between the two on the document layout.

[0041] Finally, mapping construction based on the calculated association confidence scores. A predefined confidence threshold is set. The processor iterates through all text-non-text unit pairs, and if the calculated association confidence score, i.e. or , is greater than the confidence threshold, it is determined that there is a strong association between the two, and the index of the unit pair and the corresponding association confidence score are stored as a triple, i.e. or , in the final association mapping table M .

[0042] By executing the above steps S1100 to S1300, a complete and structured cross-modal literature dataset is obtained, which is a composite data structure encapsulating multi-level information, including a content unit set , a feature vector set , and an association mapping table .

[0043] S2000, based on the association mapping table , forms a text context set for each non-text unit to construct a cross-modal task unit set , and drives the text logic processing module TLA and the visual semantic processing module VSA to perform preliminary interpretation on the cross-modal task units, respectively outputting an initial text summary and an initial information description set , and aggregating the outputs of the two modules to construct an initial summary set .

[0044] Specifically, this step aims to utilize the cross-modal literature dataset obtained in step S1000, distribute it to two parallel processing modules, i.e., the text logic processing module (Text Logic Analysis Module, TLA) and the visual semantic processing module (Visual Semantic Analysis Module, VSA), and drive them to independently generate preliminary interpretations of local cross-modal information to construct an initial summary set .

[0045] Further, step S2000 includes: Step S2100, based on the association mapping table , iterates through each non-text unit in the content unit set , queries and filters all associated text block units for the non-text unit to form a text context set, encapsulates the non-text unit and its corresponding text context set as a cross-modal task unit, and constructs a cross-modal task unit set .

[0046] Specifically, this step aims to aggregate the relevant textual context for each non-text object based on the established cross-modal associations. The collection of content units in and associated mapping tables , transformed into a set of cross-modal task units with non-text objects as the core , providing a local and complete information input environment for the subsequent two parallel processing modules. Represents a text-visual task unit, encapsulating the j a visual object and all its associated textual descriptions; Represents a text-formula task unit, encapsulating the A formula object and its associated text context.

[0047] In the specific implementation process, this step executes the task distribution function to associate the mapping table Based on, a collection of content units Each non-text unit in constructs its corresponding task unit. The core of this process is the context aggregation operation. For any given visual object unit , by querying the associated mapping table , from all text block units Filter out text block units that meet the preset conditions A subset of , forming its associated text context set The screening condition is that in the association mapping table M There is at least one triple entry in , and the text index of the triple entry is i and the non-text index is j Based on the filter conditions, a semantic association query operation is completed to extract all the visual object units related to the given There is textual content with established associations.

[0048] After completing the visual object unit Associated with its text context set After binding, the two are encapsulated into an ordered tuple to form a complete text-vision task unit Similarly, for any given formula object cell , using the same logical steps to form the corresponding text-formula task unit .in, Indicates the Formula object cell The associated text context collection.

[0049] Step S2200, drive the text logic processing module TLA to perform deep semantic understanding and information compression on the text context set associated with each non-text unit, and generate an initial text summary representing the context set .

[0050] Specifically, this step aims to drive the text logic processing module TLA to perform deep semantic understanding and information compression on the text context set associated with the non-text unit constructed in step S2100, and generate an initial text summary set that preliminarily summarizes the core content of the text context set . Among them, and respectively represent the initial text summary generated by the text logic processing module TLA for the text context set associated with the j th visual object unit and the th formula object unit at the initial time step time= 0; specifically the text logic processing module TLA; represents the initial time step time= 0.

[0051] In the specific implementation process, the text logic processing module TLA is a text generator based on a generative large language model, and its generation behavior is defined by pre-trained model parameters. The text generator receives all string contents in the text context set as input, uses an encoder-decoder architecture composed of multiple self-attention mechanisms, and combines decoding strategies such as beam search to perform deep semantic understanding and information integration, and outputs coherent and summarized initial text summaries.

[0052] Step S2300, drive the visual semantic processing module VSA to perform visual feature extraction and semantic modeling on the pixel matrix of the visual object unit and perform syntax analysis and semantic understanding on the source code content of the formula object unit according to the category label of the non-text unit, and generate corresponding initial information descriptions to construct an initial information description set .

[0053] Specifically, this step aims to drive the visual semantic processing module VSA to generate structured textual descriptions, i.e., an initial information description set based on the visual object unit or formula object unit selected in step S2100. Among them, and respectively represent the initial information description generated by the visual semantic processing module VSA for the first j visual object unit and the first formula object unit at the initial time step time= 0; specifically the visual semantic processing module VSA; representing the initial time step time= 0.

[0054] In the specific implementation process, the core of the visual semantic processing module VSA is a multi-modal large language model, which internally integrates a multi-modal content generator with input modal adaptive processing capability. The multi-modal content generator executes corresponding analysis and generation logic according to the category label of the received non-text information, as follows: First, when the visual object unit is input, the multi-modal content generator directly processes the pixel matrix contained in the visual object unit , performs feature extraction and semantic modeling through internal visual encoders and language decoders, such as identifying chart types, extracting curve trends, locating key data points, and analyzing architectural components, and outputs the initial information description generated for the first j visual object unit .

[0055] Second, when the formula object unit is input, the multi-modal content generator processes the source code content contained in the formula object unit , performs syntax analysis and semantic understanding, analyzes its mathematical meaning, such as identifying variable relationships, explaining operation logic, and explaining the main idea of the formula, and generates the initial information description generated for the first formula object unit .

[0056] Step S2400, for each cross-modal task unit, aggregate the initial text summary set generated by the text logic processing module TLA and the initial information description set generated by the visual semantic processing module VSA to construct an initial summary set .

[0057] Specifically, this step aims to aggregate the initial text summary set generated in step S2200 and the initial information description set generated in step S2300 to construct a unified and structured initial summary set To provide formalized initial observations for subsequent steps.

[0058] In a specific implementation, a state construction function is executed in this step. The state construction function, for each cross-modal task unit whose processing is completed in step S2100, combines its corresponding initial text summary and initial information description into an ordered tuple as an initial joint state. The tuple encapsulates the preliminary understanding of the cross-modal task unit before the start of reinforcement learning optimization, i.e., the initial time step time= 0.

[0059] The construction process of the initial joint state can be formalized as follows for different types of cross-modal task units: First, for a text-visual task unit , its initial joint state is constructed as .

[0060] Second, for a text-formula task unit , its initial joint state is constructed as .

[0061] where denotes the initial joint state of the time= th text-visual task unit at time step j 0, which is an ordered tuple encapsulating the core data structures of the initial text summary and the initial information description ; denotes the initial joint state of the time= th text-formula task unit at time step 0, which is an ordered tuple encapsulating the core data structures of the initial text summary and the initial information description .

[0062] Finally, the initial joint states generated by all cross-modal task units collectively constitute the initial summary set .

[0063] S3000, an iterative closed loop driven by cross-modal consistency rewards is started, a joint action is constructed according to the current joint state , a reward signal is calculated by a cross-modal consistency arbitrator CMCA according to the joint state , and the joint action is updated based on the reward signal Construct a proximal policy optimization PPO algorithm to update the policy network of the text logic processing module TLA and the visual semantic processing module VSA, complete the state update and based on the reward signal The convergence of the closed loop is used to determine whether to terminate the closed loop.

[0064] Specifically, this step aims to use the initial summary set constructed in step S2400 As the initial state, it is iteratively optimized in a multi-agent reinforcement learning (MARL) closed-loop control process driven by a cross-modal consistency reward function to output a final joint state set that is stable and consistent at the semantic level. Wherein, the reward function is a feature vector set formed in step S1200 F and the content unit set produced in step S1100 U As the basis for calculation.

[0065] Furthermore, step S3000 includes: Step S3100: drive the text logic processing module TLA and the visual semantic processing module VSA to respectively The local observation values ​​are input into their respective policy networks to generate the predefined action space The probability distribution on the , determines the corresponding action according to the probability distribution, and constructs the actions determined by the two modules into a joint action .

[0066] Specifically, this step aims to define the core decision-making mechanism for multi-agent reinforcement learning (MARL). This mechanism drives the text logic processing module TLA and the visual semantic processing module VSA, based on the current joint state. local observations, using their respective policy networks to output joint actions for improving the current summary hypothesis .in, Indicates the time= The joint state at the time of the iteration, when processing visual tasks, specifically the first j The initial joint state of the text-vision task unit ; When processing formula tasks, specifically the Initial joint state of text-formula task units .

[0067] In the specific implementation process, in each round of iteration iter In the text logic processing module TLA and the visual semantic processing module VSA are independent decision-making units, respectively, from the current joint state The content part generated by itself is extracted as a local observation value and input into the policy network dedicated to it. The local observation value of the text logic processing module TLA is the text summary in the current joint state , and the local observation value of the visual semantic processing module VSA is the information description in the current joint state .

[0068] The policy network is a deep neural network defined by trainable parameters, such as a policy evaluation network with an Actor-Critic architecture. After receiving the local observation value, the policy network outputs a probability distribution over a predefined and discrete action space . According to the probability distribution, random sampling is performed to determine the specific actions to be executed by each module in this round, i.e., the text logic processing module TLA action selection and the visual semantic processing module VSA action selection . Finally, the independent action selection and of the two modules together form an ordered tuple, i.e., the joint action of this round of iteration .

[0069] Step S3200, drive the cross-modal consistency arbitrator CMCA to evaluate the current joint state from three dimensions of semantic alignment reward , numerical consistency reward , and logical compliance reward , and obtain the reward signal by weighted sum of the reward scores of each dimension .

[0070] Specifically, this step aims to design and calculate a reward signal that can quantitatively evaluate the consistency degree between the outputs of the text logic processing module TLA and the visual semantic processing module VSA . The reward signal is the core basis for the collaborative optimization of the two modules.

[0071] In the specific implementation process, the cross-modal consistency arbitrator (CMCA) is built in this step. At the end of each iteration, the arbitrator CMCA calculates the comprehensive evaluation from the three dimensions of semantics, quantification, and structure for the current joint state , and obtains the final reward signal by weighted sum of the scores of each dimension . Specifically as follows: The first dimension is the semantic alignment reward The semantic alignment reward is used to evaluate the alignment degree of the two module outputs, i.e., the text description and the information description, at the semantic level. The arbitrator CMCA calls a sentence embedding model to map the two text descriptions into high-dimensional semantic vectors respectively, and calculates a score based on the cosine similarity as the semantic alignment reward .

[0072] The second dimension, the numerical consistency reward . The numerical consistency reward is used to verify whether the specific quantitative statements in the text description are consistent with the original non-text data. The processing logic of the arbitrator CMCA is adaptively adjusted according to the type of the non-text unit.

[0073] For visual objects, the reward is manifested as data trend verification. The arbitrator CMCA extracts a quantitative visual trend vector from the visual object unit , such as the slope of a series of key points on a curve. At the same time, the corresponding trend statement vector is parsed from the text description of the current state. The numerical consistency reward is inversely proportional to the norm error between the two vectors.

[0074] For formula objects, the reward is manifested as numerical substitution verification. The arbitrator CMCA calls a numerical parsing function to parse whether there is a specific calculation statement of the formula in the text description of the current state, such as parsing the variable assignment pair (x=2, y=5) from the text "when x=2, y=5". If it exists, a verification function is called to substitute the parsed value into the formula object unit for verification, and the numerical consistency reward is output according to the Boolean value of the verification result.

[0075] The third dimension, the logical compliance reward . The logical compliance reward is used to verify whether the structure, process or logical relationship in the text description is consistent with the original non-text data.

[0076] For visual objects, the reward is manifested as topological structure compliance verification. The arbitrator CMCA represents the topological information of the visual object unit as a reference graph structure, where the nodes represent components and the edges represent relationships. At the same time, a structure parsing function is called to extract the step and component relationship in the text description of the current state and construct a text derived graph structure. The reward value is calculated based on the graph edit distance between the two graph structures.

[0077] For formula objects, the reward is manifested as mathematical syntax structure compliance verification. The arbitrator CMCA represents the formula object unit The parse is into its abstract syntax tree as a benchmark logical tree. At the same time, a structure parsing function is called to extract the declaration of the mathematical structure of the formula from the text description of the current state, such as identifying the operation type, the number of operations, and the operation level, and constructing a text-derived explicit logical tree. The reward value is calculated according to the matching degree of the two logical trees at the key nodes and the level relationship.

[0078] Finally, the reward signal is obtained by weighting and summing the above-mentioned sub-rewards by a preset weight.

[0079] Step S3300, based on the reward signal , the advantage function is calculated, the target function of the proximal policy optimization (PPO) algorithm is constructed in combination with the action probability ratio, and the parameter set of the text logic processing module TLA and the parameter set of the policy network of the visual semantic processing module VSA are updated synchronously.

[0080] Specifically, the reward signal obtained in step S3200 is used as objective feedback to iteratively update the parameters of the text logic processing module TLA and the policy network of the visual semantic processing module VSA, and output the updated parameter set of the text logic processing module TLA policy network and the parameter set of the visual semantic processing module VSA policy network .

[0081] In the specific implementation process, a stable policy gradient algorithm, i.e., the proximal policy optimization (PPO) algorithm, is used to update the network parameters of the text logic processing module TLA and the visual semantic processing module VSA. The specific update process mainly includes the following calculation links: First, the advantage function is calculated. Based on the collected experience sequence, the advantage function of the action performed in the state is calculated. The experience sequence is composed of the joint state iter , the joint action , and the reward signal of the first round. The advantage function is used to evaluate the pros and cons of the selected action compared to the average action value in the state. A positive advantage value means that the current action is better than the average policy and should be encouraged; otherwise, it should be suppressed.

[0082] Second, target function construction and optimization. Based on the action probability ratio and the above advantage function, a target function specific to the proximal policy optimization algorithm PPO is constructed, which is obtained by calculating the mathematical expectation of a certain key indicator on the experience sequence batch. Among them, the action probability ratio is a key indicator for measuring the probability change amplitude of selecting the same action in the same state before and after the policy network update, which is composed of the probability output ratio of the new policy network executing the current optimization and the old policy network executing the sampling to collect experience data; the key indicator is determined by selecting one of the following two candidate values through the minimum value function, which realizes the conservative principle of policy update.

[0083] The first candidate value is the unconstrained policy optimization target. This item directly multiplies the action probability ratio and the advantage function, which is used to proportionally adjust the policy according to the positive and negative of the advantage function. The second candidate value is the constrained policy optimization target. This item introduces a clipping mechanism to forcibly constrain the action probability ratio within a narrow trust region defined by the hyperparameters . The second candidate value is the constrained policy optimization target. This item introduces a clipping mechanism to forcibly constrain the action probability ratio within a narrow trust region defined by the hyperparameters

[0084] Third, parameter gradient update. The stochastic gradient ascent (SGA) algorithm is used to optimize and update the unified parameter set of the policy network according to the gradient direction of the target function . The specific process formula is as follows: ; Among them, represents the unified parameter set after the iter th iteration update; represents the unified parameter set before the iter th iteration update; represents the unified parameter set of all policy networks to be optimized, including represents the parameter set of the text logic processing module TLA policy network and represents the parameter set of the visual semantic processing module VSA policy network; represents the learning rate in the parameter update process, which is used to control the step size of single parameter update; represents the gradient of the target function with respect to the unified parameter set before the iter th iteration update.

[0085] ​Finalize the unified parameter set From its initial state in the first iter iteration Evolve into the optimized state . This update will be applied synchronously to its subset, i.e. the parameter set of the text logic processing module TLA policy network and the parameter set of the visual semantic processing module VSA policy network . Wherein, the optimized state will be enabled when the flow returns to step S3100 of the next round to drive the parameter generation of the policy network to generate better actions, forming a feedback control closed loop.

[0086] Step S3400, according to the joint action , update the current joint state to generate the joint state of the next iteration, monitor the reward signal in the relative change rate in continuous iterations, if the absolute value of the relative change rate is continuously lower than the preset convergence threshold, terminate the iteration cycle and output the final joint state set , otherwise, feed back the updated joint state to the decision step to start the next iteration.

[0087] Specifically, this step aims to update the current joint state based on the joint action output by step S3100 to generate the joint state of the next iteration, while continuously monitoring the change of the reward signal to determine whether the optimization process has reached the convergence condition, so as to decide whether to continue iteration or terminate the process and output the final joint state set .

[0088] In the specific implementation process, this step includes two parallel processing logics of state transition and convergence judgment, which are as follows: First, state transition. According to the specific instructions of the joint action iter of the first iteration, update the current joint state to generate the next joint state . Wherein, and respectively represent the text summary and information description constituting the current joint state iter at the beginning of the first iteration.

[0089] If the action of any module is "correct" or "rewrite", the module will call its internal generator to generate the current joint state The corresponding description part is regenerated to obtain an updated text summary and information description as the corresponding content of the next round of joint state . Among them, and respectively represent the temporary content newly generated by the module taking the "correction" or "rewriting" action in the first iter round of iteration.

[0090] If the action of any module is "keep", the current joint state corresponding to the text summary and information description content remains unchanged.

[0091] After generating the next round of joint state , the iteration round index is incremented, and the control flow returns to step S3100 to start a new round of decision-making.

[0092] Second, convergence judgment. In order to avoid infinite iteration and ensure the stability of the output, the change of the reward signal is continuously monitored. When the absolute value of the relative change rate of the reward signal in the last several rounds of iteration is less than the preset convergence threshold, it is determined that the iteration process in the multi-agent reinforcement learning (MARL) closed-loop control process has converged, and the loop is terminated. At this time, the joint state of the current iteration round is marked as the final joint state, that is, the final joint state set .

[0093] S4000, deconstructing the final joint state set to separate the final text summary set and the final information description set , using the association mapping table M as a structured guide to drive the sequence-to-sequence language model to fuse and reconstruct the two separated sets to generate the final summary .

[0094] Specifically, this step aims to integrate the final joint state set output by step S3400 into a complete and coherent final summary through a context-aware fusion generation process, realizing the integration of fragmented information into a whole chapter that conforms to human reading habits and is logically clear.

[0095] Further, step S4000 includes: Step S4100, data reconstruction is performed on the final joint state set to extract and separate the text summary and information description encapsulated thereby, and to construct the final text summary set and final information description set .

[0096] Specifically, this step aims to reconstruct data from the final joint state set by extracting and separating the text-side summaries and non-text-side descriptions contained therein, outputting the aggregated final text summary set and final information description set .

[0097] The final joint state set is the union of the final joint state set of all text-visual task units and the final joint state set of all text-formula task units.

[0098] In specific implementation processes, a segment extraction function is used to traverse each member in the final joint state set , i.e., each final joint state . Among them, x represents a general index, which can correspond to the index j of a visual object or the index of a formula object; and represent the text summary and information description in the final joint state x indicated by the general index .

[0099] For each joint state , the segment extraction function performs unpacking operations to extract the text summary in each final joint state and add it as an element to the final text summary set . Similarly, the information description in each final joint state is extracted and added as an element to the final information description set .

[0100] Step S4200 uses the association mapping table M as a structured contextual guide to drive a sequence-to-sequence language model to perform a natural language generation task on the final text summary set and final information description set to generate a final summary .

[0101] Specifically, this step aims to reconstruct data from the final text summary set and final information description set , using the association mapping table constructed in step S1300 M As a structural guide, a generative model is used to intelligently integrate and reconstruct the text, producing a final summary with clear logic and structure that reflects the core innovations of the original text. .

[0102] In the specific implementation process, this step enables a summary fusion module, whose core is a large-scale sequence to sequence (Seq2Seq) language model fine-tuned by text rewriting and paragraph generation tasks, and its functionality is defined by the conditional fusion function.

[0103] The conditional fusion function takes all optimized summary fragments as input, i.e. the final text summary set and the final information description set , while using cross-modal association mapping tables M As a structural context condition, the chapter structure prior is obtained. The structural context specifically refers to the association mapping table M The provided text-level information goes beyond the semantics of individual sentences, namely the spatial proximity between abstract fragments and the display of citation relationships.

[0104] Based on this structured context, the fusion function performs a series of natural language generation tasks, including content planning, sentence rearrangement, cohesion and coherence generation, and citation relationship reconstruction, and finally generates a final summary that is contextually coherent and logically smooth. Among them, content planning is used to determine the order of exposition of each topic or key point, sentence rearrangement is used to organize semantically related abstract fragments into adjacent groups, cohesion and coherence generation is used to add necessary transition words and logical conjunctions between abstract fragments, and reference relationship reconstruction is used to generate reference words that conform to the logic of the original text, such as "such as iter shown".

[0105] Example 2 This embodiment provides an LLM output stability control system based on reinforcement learning on the basis of embodiment 1, such as Figure 1 Figure 2 As shown, it includes a document modality segmentation module, a dual-module parallel interpretation module, a cross-modal reward arbitration module, and a joint state deconstruction module; The document modality segmentation module is used to segment unstructured document data. Perform page layout analysis to segment and extract multimodal content regions, converting them into a collection of content units U , for the content unit set U Each content unit in the feature vector is vectorized to generate a feature vector set and construct the cross-modal correlation mapping between the content units to construct a correlation mapping table M , the literature data is converted into a cross-modal literature dataset containing a content unit set U , a feature vector set F and a correlation mapping table M ; The double-module parallel reading module is used to form a text context set from each non-text unit according to the correlation mapping table to construct a cross-modal task unit set , and the text logic processing module TLA and the visual semantic processing module VSA are driven in parallel to perform preliminary reading on the cross-modal task units, and an initial text summary and an initial information description set are output respectively, the outputs of the two modules are aggregated to construct an initial summary set ; The cross-modal reward arbitration module is used to start an iterative closed loop driven by a cross-modal consistency reward, construct a joint action according to the current joint state , calculate a reward signal by a cross-modal consistency arbitrator CMCA according to the joint state , construct a proximal policy optimization PPO algorithm based on the reward signal to update the policy network of the text logic processing module TLA and the visual semantic processing module VSA, complete state update and determine whether to terminate the closed loop according to the convergence of the reward signal ; The joint state deconstruction module is used to deconstruct the final joint state set to separate the final text summary set and the final information description set , and drive the sequence-to-sequence language model to fuse and reconstruct the two separated sets under the guidance of the correlation mapping table M to generate a final summary .

[0106] The above technical solutions provided in the embodiments of the present application have not been described in detail, so as not to be too repetitive.

[0107] ​The specific embodiments described above are further explained in connection with the purposes, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. The LLM output stability control method based on reinforcement learning is characterized by: include: performing page layout analysis on unstructured document data to segment and extract multimodal content areas, converting the content areas into content unit sets, performing feature vectorization on each content unit in the content unit set to generate a feature vector set, constructing a cross-modal association mapping between the content units to construct an association mapping table, and converting the document data into a cross-modal document dataset comprising the content unit set, the feature vector set, and the association mapping table; According to the association mapping table, each non-text unit is formed into a text context set to construct a cross-modal task unit set. The text logic processing module and the visual semantic processing module are driven in parallel to perform preliminary interpretation of each cross-modal task unit, outputting an initial text summary and an initial information description set respectively. The outputs of the two modules are aggregated to construct an initial summary set. Initiate an iterative closed loop driven by cross-modal consistency rewards, construct a joint action based on the current joint state, calculate a reward signal based on the joint state by a cross-modal consistency arbitrator, construct a proximal policy optimization algorithm based on the reward signal to update the policy networks of the text logic processing module and the visual semantic processing module, complete the state update, and determine whether to terminate the closed loop based on the convergence of the reward signal; The final joint state set is deconstructed to separate the final text summary set and the final information description set. Using the association mapping table as a structural guide, the sequence-to-sequence language model is driven to fuse and reconstruct the two separated sets to generate the final summary.

2. The LLM output stability control method based on reinforcement learning according to claim 1 is characterized in that: The conversion steps of the cross-modal document dataset include: Performing page layout analysis and semantic segmentation on unstructured document data, identifying and extracting text, image, and formula content areas with category labels and bounding boxes, and converting the extracted content areas into a structured content unit set; Based on the category label of each content unit in the content unit set, the corresponding text or visual encoder based on the Transformer architecture is called to encode the content unit into a corresponding feature vector, forming a feature vector set containing text, visual and formula semantics; For each pair of text block units and non-text units in the content unit set, a comprehensive association scoring algorithm is executed to calculate an explicit reference relationship score and a spatial proximity score, respectively, and the two scores are weighted to generate an association confidence score. The combination pairs whose association confidence scores exceed a preset confidence threshold are screened out, and an association mapping table is constructed. By executing all the above processes, a cross-modal document dataset is obtained.

3. The LLM output stability control method based on reinforcement learning according to claim 1, characterized in that: The method for constructing the initial summary set includes: Based on the association mapping table, traverse each non-text unit in the content unit set, query and filter all associated text block units for the non-text unit to form a text context set, encapsulate each non-text unit and its corresponding text context set into a cross-modal task unit, and construct a cross-modal task unit set; Drive the text logic processing module to perform deep semantic understanding and information compression on the text context set associated with each non-text unit, and generate an initial text summary representing the context set; Drive the visual semantic processing module to perform visual feature extraction and semantic modeling on the pixel matrix of the visual object unit according to the category label of the non-text unit, and perform syntax parsing and semantic understanding on the source code content of the formula object unit, generate the corresponding initial information description, and construct the initial information description set; For each cross-modal task unit, the initial text summary set generated by the text logic processing module and the initial information description set generated by the visual semantic processing module are aggregated to construct an initial summary set.

4. The LLM output stability control method based on reinforcement learning according to claim 3 is characterized in that: The screening method of the text context set includes at least one triple entry in the association mapping table, the triple entry has a corresponding text index and a corresponding non-text index, and based on the screening method, a semantic association query operation is completed to extract all text contents that have a predetermined association with a given visual object unit.

5. The LLM output stability control method based on reinforcement learning according to claim 1, characterized in that: The updating steps of the policy networks of the text logic processing module and the visual semantic processing module include: The text logic processing module and the visual semantic processing module are driven to input the local observation values ​​of the current joint state into their respective policy networks, generate a probability distribution on a predefined action space, determine the corresponding action based on the probability distribution, and construct the actions determined by the two modules into a joint action; Drive the cross-modal consistency arbitrator to comprehensively evaluate the current joint state from three dimensions: semantic alignment reward, numerical consistency reward, and logical conformance reward. The reward signal is obtained by weighted summing the reward scores of each dimension. The advantage function is calculated based on the reward signal, and the objective function of the proximal policy optimization algorithm is constructed in combination with the action probability ratio. The stochastic gradient ascent algorithm is used to synchronously update the parameter set of the text logic processing module and the parameter set of the policy network of the visual semantic processing module along the gradient direction of the objective function.

6. The LLM output stability control method based on reinforcement learning according to claim 5, characterized in that: The semantic alignment reward is used to evaluate the degree of alignment between the text summary and the information description at the semantic level; the numerical consistency reward is used to verify whether the specific quantitative statements in the text description are consistent with the original non-text data; and the logical consistency reward is used to verify whether the structure, process or logical relationship in the text description is consistent with the original non-text data.

7. The LLM output stability control method based on reinforcement learning according to claim 1, characterized in that: The method for determining the termination of the closed loop includes: updating the current joint state based on the joint action to generate the joint state for the next iteration, monitoring the relative change rate of the reward signal in consecutive iterations, and terminating the iterative loop to output the final joint state set if the absolute value of the relative change rate is continuously lower than a preset convergence threshold; otherwise, feeding the updated joint state back to the decision step to start the next iteration.

8. The LLM output stability control method based on reinforcement learning according to claim 1, characterized in that: The method for generating the final summary includes: Reconstruct the data of the final joint state set, extract and separate the encapsulated text summary and information description, and construct the final text summary set and final information description set; Using the association mapping table as a structured context guide, the sequence-to-sequence language model is driven to perform the natural language generation task on the final text summary set and the final information description set to generate the final summary.

9. An LLM output stability control system based on reinforcement learning, which is used to implement the LLM output stability control method based on reinforcement learning according to any one of claims 1 to 8, characterized in that: The system includes a document modality segmentation module, a dual-module parallel interpretation module, a cross-modal reward arbitration module, and a joint state deconstruction module; The document modality segmentation module is configured to perform page layout analysis on unstructured document data to segment and extract multimodal content regions, convert the content regions into content unit sets, perform feature vectorization on each content unit in the content unit set to generate a feature vector set, construct a cross-modal association mapping between the content units to construct an association mapping table, and convert the document data into a cross-modal document dataset comprising a content unit set, a feature vector set, and an association mapping table; The dual-module parallel interpretation module is used to form a text context set for each non-text unit according to the association mapping table to construct a cross-modal task unit set, and drive the text logic processing module and the visual semantic processing module in parallel to perform preliminary interpretation on each cross-modal task unit, output an initial text summary and an initial information description set respectively, and aggregate the outputs of the two modules to construct an initial summary set; The cross-modal reward arbitration module is used to initiate an iterative closed loop driven by a cross-modal consistency reward, construct a joint action based on the current joint state, calculate a reward signal based on the joint state by a cross-modal consistency arbitrator, construct a proximal policy optimization algorithm based on the reward signal to update the policy networks of the text logic processing module and the visual semantic processing module, complete the state update, and determine whether to terminate the closed loop based on the convergence of the reward signal; The joint state deconstruction module is used to deconstruct the final joint state set to separate the final text summary set and the final information description set, and drive the sequence-to-sequence language model to fuse and reconstruct the two separated sets based on the association mapping table to generate the final summary.

Images