MO-KTO legal model enhancement method, device and equipment and storage medium

Abstract

The invention discloses an MO-KTO law model enhancement method, device and equipment and a storage medium, and the method comprises the steps: obtaining a plurality of law-related optimization targets, and generating a multi-bit binary signal tag according to the performance of an original law text sample in each optimization target; inputting the law big language model, the multi-bit binary signal data labeled according to the multi-bit binary signal label and each optimization target into an MO-KTO algorithm framework to obtain training process data and an optimized target law big language model; performing multi-dimensional capability evaluation on the target law big language model by using a preset law test set to obtain performance indexes under each optimization target, and judging whether the target law big language model is successfully enhanced or not; performance reduction caused by target conflicts in traditional multi-target reinforcement learning can be effectively avoided; it is ensured that key law dimensions are remarkably improved in a balanced mode, and therefore it is accurately verified that enhancement success of the target law large language model is achieved.

Description

Technical Field

[0001] This invention relates to the field of multi-objective reinforcement learning optimization technology, and in particular to a method, apparatus, device and storage medium for enhancing a MO-KTO legal model. Background Technology

[0002] After being built, large language models establish basic language comprehension and representation structures through a pre-training phase, and enhance their capabilities in specific domains through supervised learning. However, when faced with certain specific human preference goals that are difficult to quantify or visualize, reinforcement learning algorithms are mostly used to train the large model. In multi-objective reinforcement algorithms for legal large models, the common approach is to collect multi-preference data from different dimensions, train a separate reward model for each goal, and then integrate all reward models by assigning different weights to form a composite reward model for reinforcement learning. However, there are conflicts between the multi-dimensional reinforcement goals in the legal field, and performing reinforcement learning on multiple goals simultaneously is very unstable. Summary of the Invention

[0003] The main objective of this invention is to provide a method, apparatus, device, and storage medium for enhancing MO-KTO legal models, aiming to solve the technical problems of performance degradation caused by objective conflicts and complex annotation processes in the multi-objective optimization of large legal language models in the prior art.

[0004] In a first aspect, the present invention provides a method for enhancing an MO-KTO legal model, the method comprising the following steps: Obtain several law-related optimization objectives, and generate multi-bit binary signal labels based on the performance of the original legal text samples on each optimization objective; The legal big language model is input into the MO-KTO algorithm framework based on the multi-bit binary signal data labeled with the multi-bit binary signal tags and each optimization target to obtain training process data. The legal big language model is then optimized based on the training process data to obtain the optimized target legal big language model. The target legal language model is evaluated in multiple dimensions using a preset legal test set to obtain performance indicators under each optimization objective. Based on the performance indicators, it is determined whether the target legal language model has been successfully enhanced.

[0005] Optionally, the step of obtaining several legally related optimization objectives, and generating multi-bit binary signal tags based on the performance of the original legal text samples on each optimization objective, includes: The optimization goals are to obtain the accuracy of the corresponding legal provisions, the logical coherence of the judgment, the semantic compliance, and the fairness. Binary signal labels for each optimization objective are generated based on the actual performance of the original legal text sample on each optimization objective.

[0006] Optionally, the step of inputting the legal big language model, based on the multi-bit binary signal data labeled with the multi-bit binary signal tags and each optimization objective, into the MO-KTO algorithm framework to obtain training process data, and optimizing the legal big language model based on the training process data to obtain the optimized target legal big language model, includes: The labeled multi-bit binary signal data is determined based on the multi-bit binary signal label; The legal language model, the multi-bit binary signal data, and each optimization objective are input into the MO-KTO algorithm framework; The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism, and the loss function is calculated in real time. The parameters of the legal big language model are updated according to the loss function to obtain training process data. The legal big language model is then optimized based on the training process data to obtain the optimized target legal big language model.

[0007] Optionally, the step of using the MO-KTO algorithm framework to drive multi-objective reinforcement learning through a binary signal alignment mechanism and calculating the loss function in real time includes: The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism to obtain the value function and the decision weights of each optimization objective; A loss function is dynamically constructed based on the value function and the decision weights.

[0008] Optionally, the step of using the MO-KTO algorithm framework to drive multi-objective reinforcement learning through a binary signal alignment mechanism to obtain the value function and the decision weights of each optimization objective includes: The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism. The value function and decision weights for each optimization objective are obtained using the following formula:

[0009]

[0010]

[0011] in, For quantification samples In the The value function of an optimization objective The weighting coefficients for the revenue side. The weighting coefficients for the loss side. It is the Sigmoid smoothing function. For the first The reward value for each optimization objective. For the first The baseline reference value for each optimization objective. For input Below, output Compliant with legal regulations For input Below, output Violation of laws and regulations For hyperparameters, For the first The decision weight of each legal optimization objective. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability of.

[0012] Optionally, the step of dynamically constructing a loss function based on the value function and the decision weights includes: Based on the value function and the decision weights, a loss function is dynamically constructed using the following formula:

[0013] in, Let MO-KTO be the total loss function of the algorithm. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability, Summing all optimization objectives, In the dataset Expected value on Represents the weight vector transpose, For the first The decision weight of each legal optimization objective. For quantification samples In the The value function of an optimization objective.

[0014] Optionally, the step of using a preset legal test set to perform a multi-dimensional capability evaluation of the target legal language model, obtaining performance indicators under each optimization objective, and determining whether the target legal language model has been successfully enhanced based on the performance indicators includes: The target legal language model is evaluated in multiple dimensions using a preset legal test set. Independent quantitative performance indicators for each optimization objective are calculated, and evaluation results are generated. Based on the evaluation results, Pareto front curves are plotted, and the trade-offs and non-dominated solution sets among the optimization objectives are dynamically visualized based on the Pareto front curves. When the performance improvement of the preset key legal objectives among the optimization objectives is detected, and all indicators of the preset key legal objectives reach the preset legal professional threshold, the target legal language model is determined to be successfully enhanced.

[0015] Secondly, to achieve the above objectives, the present invention also proposes an MO-KTO legal model enhancement device, the MO-KTO legal model enhancement device comprising: The tag generation module is used to obtain several legally related optimization objectives and generate multi-bit binary signal tags based on the performance of the original legal text sample on each optimization objective. The model optimization module is used to input the legal big language model, the multi-bit binary signal data labeled with the multi-bit binary signal labels and each optimization target into the MO-KTO algorithm framework to obtain training process data, and optimize the legal big language model based on the training process data to obtain the optimized target legal big language model. The judgment module is used to perform multi-dimensional capability evaluation on the target legal language model using a preset legal test set, obtain performance indicators under each optimization objective, and judge whether the target legal language model has been successfully enhanced based on the performance indicators.

[0016] Thirdly, to achieve the above objectives, the present invention also proposes a MO-KTO legal model enhancement device, the MO-KTO legal model enhancement device comprising: a memory, a processor, and a MO-KTO legal model enhancement program stored in the memory and executable on the processor, the MO-KTO legal model enhancement program being configured to implement the steps of the MO-KTO legal model enhancement method as described above.

[0017] Fourthly, to achieve the above objectives, the present invention also proposes a storage medium storing a MO-KTO legal model enhancement program, wherein the MO-KTO legal model enhancement program, when executed by a processor, implements the steps of the MO-KTO legal model enhancement method as described above.

[0018] The MO-KTO legal model enhancement method proposed in this invention obtains several law-related optimization objectives and generates multi-bit binary signal labels based on the performance of the original legal text samples on each optimization objective. The legal language model, labeled with the multi-bit binary signal data and each optimization objective, is input into the MO-KTO algorithm framework to obtain training process data. The legal language model is then optimized based on this training data to obtain an optimized target legal language model. A preset legal test set is used to evaluate the target legal language model's capabilities across multiple dimensions, obtaining performance indicators under each optimization objective. These performance indicators determine whether the target legal language model has been successfully enhanced. This method effectively avoids the performance degradation caused by objective conflict in traditional multi-objective reinforcement learning, ensuring significant and balanced improvements in key legal dimensions. This accurately verifies the successful enhancement of the target legal language model, effectively alleviates multi-objective conflict, improves the overall model performance, significantly enhances the stability and efficiency of reinforcement learning, reduces data labeling costs, simplifies data preparation processes, refines data alignment attributes, improves model learning accuracy, and enhances the model's adaptability to complex legal scenarios. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the first embodiment of the MO-KTO legal model enhancement method of the present invention; Figure 3 This is a flowchart illustrating the second embodiment of the MO-KTO legal model enhancement method of the present invention; Figure 4 This is a flowchart illustrating the third embodiment of the MO-KTO legal model enhancement method of the present invention; Figure 5 Here is a diagram of the MO-RLHF algorithm framework; Figure 6 Here is a diagram of the MO-DPO algorithm framework; Figure 7 Flowchart for KTO and DPO training; Figure 8 This is a schematic diagram of the MO-KTO algorithm in the MO-KTO legal model enhancement method of the present invention; Figure 9 This is a flowchart illustrating the fourth embodiment of the MO-KTO legal model enhancement method of the present invention; Figure 10 MO-KTO Legal Enhancement System Architecture Diagram; Figure 11 This is a functional block diagram of the first embodiment of the MO-KTO legal model enhancement device of the present invention.

[0020] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0022] The solution of this invention mainly involves: acquiring several law-related optimization objectives; generating multi-bit binary signal labels based on the performance of the original legal text samples at each optimization objective; inputting the legal large language model, labeled with the multi-bit binary signal data and each optimization objective, into the MO-KTO algorithm framework to obtain training process data; optimizing the legal large language model based on the training process data to obtain the optimized target legal large language model; using a preset legal test set to perform multi-dimensional capability evaluation on the target legal large language model to obtain performance indicators under each optimization objective; and judging based on the performance indicators. Whether the target legal language model is successfully enhanced; effectively avoids the performance degradation caused by target conflict in traditional multi-objective reinforcement learning; ensures significant and balanced improvement in key legal dimensions, thereby accurately verifying the success of the target legal language model enhancement, effectively alleviating multi-objective conflict, improving the overall performance of the model, significantly improving the stability and efficiency of reinforcement learning, reducing data annotation costs, simplifying the data preparation process, finely characterizing data alignment attributes, improving model learning accuracy, enhancing the model's adaptability to complex legal scenarios, and solving the technical problems of performance degradation caused by target conflict and complex annotation process in the multi-objective optimization of legal language models in the prior art.

[0023] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0024] like Figure 1As shown, the device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0025] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0026] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating device, a network communication module, a user interface module, and an MO-KTO legal model enhancement program.

[0027] The device of the present invention calls the MO-KTO legal model enhancement program stored in the memory 1005 through the processor 1001 and performs the following operations: Obtain several law-related optimization objectives, and generate multi-bit binary signal labels based on the performance of the original legal text samples on each optimization objective; The legal big language model is input into the MO-KTO algorithm framework based on the multi-bit binary signal data labeled with the multi-bit binary signal tags and each optimization target to obtain training process data. The legal big language model is then optimized based on the training process data to obtain the optimized target legal big language model. The target legal language model is evaluated in multiple dimensions using a preset legal test set to obtain performance indicators under each optimization objective. Based on the performance indicators, it is determined whether the target legal language model has been successfully enhanced.

[0028] The device of the present invention, through processor 1001 calling the MO-KTO legal model enhancement program stored in memory 1005, also performs the following operations: The optimization goals are to obtain the accuracy of the corresponding legal provisions, the logical coherence of the judgment, the semantic compliance, and the fairness. Binary signal labels for each optimization objective are generated based on the actual performance of the original legal text sample on each optimization objective.

[0029] The device of the present invention, through processor 1001 calling the MO-KTO legal model enhancement program stored in memory 1005, also performs the following operations: The labeled multi-bit binary signal data is determined based on the multi-bit binary signal label; The legal language model, the multi-bit binary signal data, and each optimization objective are input into the MO-KTO algorithm framework; The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism, and the loss function is calculated in real time. The parameters of the legal big language model are updated according to the loss function to obtain training process data. The legal big language model is then optimized based on the training process data to obtain the optimized target legal big language model.

[0030] The device of the present invention, through processor 1001 calling the MO-KTO legal model enhancement program stored in memory 1005, also performs the following operations: The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism to obtain the value function and the decision weights of each optimization objective; A loss function is dynamically constructed based on the value function and the decision weights.

[0031] The device of the present invention, through processor 1001 calling the MO-KTO legal model enhancement program stored in memory 1005, also performs the following operations: The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism. The value function and decision weights for each optimization objective are obtained using the following formula:

[0032]

[0033]

[0034] in, For quantification samples In the The value function of an optimization objective The weighting coefficients for the revenue side. The weighting coefficients for the loss side. It is the Sigmoid smoothing function. For the first The reward value for each optimization objective. For the first The baseline reference value for each optimization objective. For input Below, output Compliant with legal regulations For input Below, output Violation of laws and regulations For hyperparameters, For the first The decision weight of each legal optimization objective. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability of.

[0035] The device of the present invention, through processor 1001 calling the MO-KTO legal model enhancement program stored in memory 1005, also performs the following operations: Based on the value function and the decision weights, a loss function is dynamically constructed using the following formula:

[0036] in, Let MO-KTO be the total loss function of the algorithm. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability, Summing all optimization objectives, In the dataset Expected value on Represents the weight vector transpose, For the first The decision weight of each legal optimization objective. For quantification samples In the The value function of an optimization objective.

[0037] The device of the present invention, through processor 1001 calling the MO-KTO legal model enhancement program stored in memory 1005, also performs the following operations: The target legal language model is evaluated in multiple dimensions using a preset legal test set. Independent quantitative performance indicators for each optimization objective are calculated, and evaluation results are generated. Based on the evaluation results, Pareto front curves are plotted, and the trade-offs and non-dominated solution sets among the optimization objectives are dynamically visualized based on the Pareto front curves. When the performance improvement of the preset key legal objectives among the optimization objectives is detected, and all indicators of the preset key legal objectives reach the preset legal professional threshold, the target legal language model is determined to be successfully enhanced.

[0038] This embodiment, through the above-described scheme, obtains several legally related optimization objectives. Based on the performance of the original legal text samples in each optimization objective, it generates multi-bit binary signal labels. The legal language model, labeled with these multi-bit binary signal data and each optimization objective, is input into the MO-KTO algorithm framework to obtain training process data. The legal language model is then optimized based on this training data to obtain an optimized target legal language model. A preset legal test set is used to evaluate the target legal language model's capabilities across multiple dimensions, obtaining performance indicators under each optimization objective. These performance indicators are then used to determine whether the target legal language model has been successfully enhanced. This approach effectively avoids performance degradation caused by objective conflict in traditional multi-objective reinforcement learning. It ensures significant and balanced improvements in key legal dimensions, thereby accurately verifying the successful enhancement of the target legal language model. This effectively alleviates multi-objective conflict, improves the overall model performance, significantly enhances the stability and efficiency of reinforcement learning, reduces data labeling costs, simplifies data preparation processes, refines data alignment attributes, improves model learning accuracy, and enhances the model's adaptability to complex legal scenarios.

[0039] Based on the above hardware structure, an embodiment of the MO-KTO legal model enhancement method of the present invention is proposed.

[0040] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the MO-KTO legal model enhancement method of the present invention.

[0041] In a first embodiment, the MO-KTO legal model enhancement method includes the following steps: Step S10: Obtain several law-related optimization objectives, and generate multi-bit binary signal labels based on the performance of the original legal text sample in each optimization objective.

[0042] It should be noted that by defining several law-related optimization objectives, multi-bit binary signal labels corresponding one-to-one with the optimization objectives can be generated based on the actual performance of the original legal text samples on each objective. This provides an efficient labeling basis for multi-objective reinforcement learning without the need for comparison between samples, significantly simplifies the data preparation process, and ensures accurate alignment of the legal dimensions.

[0043] Step S20: Input the legal big language model into the MO-KTO algorithm framework according to the multi-bit binary signal data labeled with the multi-bit binary signal tags and each optimization target to obtain training process data. Optimize the legal big language model according to the training process data to obtain the optimized target legal big language model.

[0044] It should be understood that the legal language model, along with labeled multi-bit binary signal data and various optimization objectives, is input into the MO-KTO algorithm framework to train the process data; subsequently, the legal language model is optimized based on the training process data to finally obtain the optimized target legal language model.

[0045] Step S30: Use a preset legal test set to perform a multi-dimensional capability evaluation of the target legal language model, obtain performance indicators under each optimization objective, and determine whether the target legal language model has been successfully enhanced based on the performance indicators.

[0046] Understandably, the target legal language model is systematically evaluated in multiple dimensions using a pre-set legal test set, and the quantitative performance indicators of each optimization objective are calculated independently. Then, the performance indicators are used to determine whether the target legal language model has been successfully enhanced.

[0047] This embodiment, through the above-described scheme, obtains several legally related optimization objectives. Based on the performance of the original legal text samples in each optimization objective, it generates multi-bit binary signal labels. The legal language model, labeled with these multi-bit binary signal data and each optimization objective, is input into the MO-KTO algorithm framework to obtain training process data. The legal language model is then optimized based on this training data to obtain an optimized target legal language model. A preset legal test set is used to evaluate the target legal language model's capabilities across multiple dimensions, obtaining performance indicators under each optimization objective. These performance indicators are then used to determine whether the target legal language model has been successfully enhanced. This approach effectively avoids performance degradation caused by objective conflict in traditional multi-objective reinforcement learning. It ensures significant and balanced improvements in key legal dimensions, thereby accurately verifying the successful enhancement of the target legal language model. This effectively alleviates multi-objective conflict, improves the overall model performance, significantly enhances the stability and efficiency of reinforcement learning, reduces data labeling costs, simplifies data preparation processes, refines data alignment attributes, improves model learning accuracy, and enhances the model's adaptability to complex legal scenarios.

[0048] Furthermore, Figure 3 This is a flowchart illustrating the second embodiment of the MO-KTO legal model enhancement method of the present invention, as shown below. Figure 3As shown, based on the first embodiment, a second embodiment of the MO-KTO legal model enhancement method of the present invention is proposed. In this embodiment, step S10 specifically includes the following steps: Step S11: Obtain the optimization objectives for the accuracy of the corresponding legal citations, the logical coherence of the judgment, the semantic compliance, and the fairness.

[0049] It should be noted that, before training, the core legal optimization objectives are predefined: accuracy of legal citation (i.e., ensuring that legal citations are accurate and error-free), logical coherence of judgments (i.e., ensuring that the reasoning chain of judgment documents is rigorous and reasonable), semantic compliance (i.e., ensuring that the generated content conforms to legal semantic norms), and fairness (i.e., avoiding legal bias in the model output). These objectives serve as the benchmark dimensions for multi-objective reinforcement learning, guiding the subsequent generation of multi-bit binary signal labels and the optimization direction of the MO-KTO algorithm. This ensures that the legal language model achieves accurate and balanced synergistic improvement in key legal professional dimensions.

[0050] Step S12: Generate binary signal labels for each optimization objective based on the actual performance of the original legal text sample on each optimization objective.

[0051] Understandably, for each legal optimization target, it is possible to directly determine whether the actual performance of the original legal text sample on that target conforms to the legal norms and standards, thereby generating an independent binary signal bit for each target, forming a multi-bit binary signal label. This completely avoids the complex labeling process of pairwise comparison between samples in traditional methods, significantly reduces data preparation costs, and provides accurate optimization input basis for the MO-KTO algorithm.

[0052] This embodiment, through the above-described scheme, obtains optimization objectives for the accuracy of corresponding legal citations, logical coherence of judgments, semantic compliance, and fairness. Based on the actual performance of the original legal text samples on each optimization objective, binary signal labels for each optimization objective are generated. This completely eliminates the complex labeling process that requires pairwise comparisons between samples in traditional multi-objective reinforcement learning, significantly reducing data preparation costs. It also provides accurate and efficient optimization input for the MO-KTO algorithm, ensuring that the legal language model achieves synergistic improvement and precise alignment of indicators such as the accuracy of legal citations and logical coherence of judgments in key legal dimensions.

[0053] Furthermore, Figure 4 This is a flowchart illustrating the third embodiment of the MO-KTO legal model enhancement method of the present invention, as shown below. Figure 4 As shown, based on the first embodiment, a third embodiment of the MO-KTO legal model enhancement method of the present invention is proposed. In this embodiment, step S20 specifically includes the following steps: Step S21: Determine the labeled multi-bit binary signal data based on the multi-bit binary signal label.

[0054] It should be noted that by structuring and organizing multi-bit binary signal labels, a standardized labeled dataset can be formed. This dataset is stored in binary vector form and directly serves as the input for the MO-KTO algorithm. This ensures that the legal language model can achieve multi-objective collaborative optimization based on a precise binary signal alignment mechanism during training. At the same time, it completely avoids the complex labeling process of pairwise comparison between samples in traditional methods, significantly improving data preparation efficiency and model optimization accuracy.

[0055] Step S22: Input the legal big language model, the multi-bit binary signal data, and each optimization objective into the MO-KTO algorithm framework.

[0056] It is understandable that the legal language model, along with labeled multi-bit binary signal data (the state of each sample in terms of the accuracy of legal citation, the logical coherence of judgment, etc., represented in binary vector form) and preset legal optimization objectives (such as the accuracy of legal citation and the logical coherence of judgment), are synchronously input into the MO-KTO algorithm framework. This efficiently drives the legal language model to achieve collaborative optimization in multiple core legal dimensions, significantly improving key indicators such as the accuracy of legal citation and the logical coherence of judgment.

[0057] Step S23: Using the MO-KTO algorithm framework, multi-objective reinforcement learning is driven by a binary signal alignment mechanism to calculate the loss function in real time.

[0058] It should be understood that by using the MO-KTO algorithm framework and a binary signal alignment mechanism, legal optimization objectives such as the accuracy of legal citations and the logical coherence of judgments are mapped to independent binary signal bits. This completely eliminates the complex labeling process that requires pairwise comparisons between samples in traditional multi-objective reinforcement learning, thereby driving multi-objective reinforcement learning and calculating the loss function in real time during the training process.

[0059] Furthermore, step S23 specifically includes the following steps: The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism to obtain the value function and the decision weights of each optimization objective; A loss function is dynamically constructed based on the value function and the decision weights.

[0060] Understandably, multi-bit binary signal labels are used to directly drive multi-objective reinforcement learning, calculate the value function of each optimization objective in real time, and obtain the decision weight of each objective. Subsequently, a loss function is dynamically constructed, and the value function is combined with the decision weight. Through a weighted aggregation mechanism, the legal language model is ensured to adaptively adjust the optimization direction of each objective during training, thereby achieving synergistic improvement of key legal dimensions such as the accuracy of legal citation and the logical coherence of judgments, and avoiding the performance degradation caused by objective conflicts in traditional methods.

[0061] Furthermore, the step utilizes the MO-KTO algorithm framework to drive multi-objective reinforcement learning through a binary signal alignment mechanism to obtain the value function and the decision weights of each optimization objective, specifically including the following steps: The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism. The value function and decision weights for each optimization objective are obtained using the following formula:

[0062]

[0063]

[0064] in, For quantification samples In the The value function of an optimization objective The weighting coefficients for the revenue side. The weighting coefficients for the loss side. It is the Sigmoid smoothing function. For the first The reward value for each optimization objective. For the first The baseline reference value for each optimization objective. For input Below, output Compliant with legal regulations For input Below, output Violation of laws and regulations For hyperparameters, For the first The decision weight of each legal optimization objective. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability of.

[0065] Furthermore, the step of dynamically constructing a loss function based on the value function and the decision weights specifically includes the following steps: Based on the value function and the decision weights, a loss function is dynamically constructed using the following formula:

[0066] in, Let MO-KTO be the total loss function of the algorithm. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability, Summing all optimization objectives, In the dataset Expected value on Represents the weight vector transpose, For the first The decision weight of each legal optimization objective. For quantification samples In the The value function of an optimization objective.

[0067] In practical implementation, the mainstream multi-objective reinforcement learning algorithms for large language models currently include: Multi-Objective Reinforcement Learning from Human Feedback (MO-RLHF), Multi-Objective Direct Preference Optimization (MO-DPO), Direct Preference Optimization (DPO), and Knowledge Transfer Optimization (KTO). For responses to legal large language models, there are various human preferences, including legal accuracy, logical rigor, linguistic standardization, conciseness, and readability. To enhance preference objectives, the following algorithms are employed... express.

[0068] See Figure 5 , Figure 5 The MO-RLHF algorithm framework diagram is as follows: Figure 5 As shown, the MO-RLHF algorithm framework includes the following core components: MO-RLHF(previous): Represents the initial version of the algorithm, used as an optimization baseline to compare the performance differences of the model before and after optimization.

[0069] Binary Cross Entropy (BCE) is a standard loss function used in machine learning for binary classification problems. It is applied three times in the framework, corresponding to the binary classification loss calculation of three legal optimization objectives: accuracy of legal citation, logical coherence of judgment, and semantic compliance. It is used to evaluate the degree to which the sample satisfies each objective ('1' indicates that the objective is satisfied, and '0' indicates that it is not satisfied).

[0070] Reward Model (RM): It includes three independent reward models (RMr1, RMr2, RMr3, etc.), which are responsible for evaluating the reward value of different legal optimization objectives. RMr1, RMr2, and RMr3 are specific examples that deal with three key legal objectives respectively.

[0071] : Represents the weighted sum of three weight parameters, used to dynamically balance the importance of different legal objectives in multi-objective optimization, and to ensure that key indicators such as the accuracy of legal citations and the logical coherence of judgments are optimized in a coordinated manner.

[0072] Reinforcement learning: The reinforcement learning process is carried out through sampled legal texts, using the text generated by the legal language model as input for multi-objective optimization.

[0073] Language Model Policy (LM policy): Based on legal professional norms, the generated content is rewarded with labels, and these reward signals are used to optimize the language model policy (LM policy) to achieve synergistic improvement in legal dimensions such as the accuracy of legal citations and the logical coherence of judgments.

[0074] exist Figure 5 In this process, a multidimensional dataset is prepared based on different legal enhancement objectives. Train a set of reward models Each model represents a different alignment target, expressed through a weight vector. Different reward models are linearly combined to form a composite training objective, and the ultimate training goal is to obtain the Pareto front of the policy model.

[0075] See Figure 6 , Figure 6 The MO-DPO algorithm framework diagram is as follows: Figure 6As shown, the MO-DPO (Multi-Objective Direct Preference Optimization) algorithm framework includes the following core components: MO-DPO(previous): Represents the initial baseline version of the MO-DPO algorithm, serving as a reference for the model before optimization and used for comparison and evaluation with the optimized model.

[0076] D1: The first dedicated dataset for the legal optimization objective (such as the accuracy of legal citations), containing labeled legal samples, used to train and evaluate the model performance for this objective.

[0077] D2: A dedicated dataset for the second legal optimization objective (such as the logical coherence of judgments), specifically designed for the optimization and evaluation of this objective.

[0078] D3: A dedicated dataset for the third legal optimization objective (such as semantic compliance) to ensure accurate optimization of the model in this legal dimension.

[0079] BCE (Binary Cross Entropy): A binary cross-entropy loss function used to calculate the binary classification loss for each legal optimization objective (such as the accuracy of legal citations and the logical coherence of judgments), where: , Weighting parameters are used to dynamically balance the importance of different legal objectives in multi-objective optimization (such as the weight of the accuracy of legal citation). Weight of logical coherence in judgment This ensures that key legal dimensions are optimized in a coordinated manner.

[0080] RM r1: Reward Model (RM) for optimizing objectives, used to evaluate the performance of a model on a specific legal objective. For example, RM r1 is used to evaluate the accuracy of legal citations.

[0081] Margin RM r2: This is the r2th legal optimization objective in the Margin Reward Model (Margin RM). It enhances the stability and robustness of the optimization by introducing a margin mechanism.

[0082] BCE with margin: This is an improved version of BCE, which is a binary cross-entropy loss function with margin. By introducing a margin parameter, it ensures that the model focuses not only on whether the objective is satisfied, but also on the "degree" of satisfaction during the optimization process, thus avoiding overfitting caused by simple binary decision-making.

[0083] LM policy: Language Model Policy refers to the target legal language model strategy optimized by the MO-DPO algorithm, which can simultaneously improve multiple dimensions such as the accuracy of legal citation and the logical coherence of judgments, and generate text that conforms to legal professional norms.

[0084] See Figure 6 The difference between MO-RLHF and MO-DPO is that MO-DPO focuses on improving DPO at the lowest cost to achieve multi-objective alignment. This is achieved by folding the weighted combination of multiple legal enhancement objectives and the training of the language model into the training process of the reward function. The final implementation is to add an extra boundary reward value to the loss function of DPO.

[0085] See Figure 7 , Figure 7 The training flowchart for KTO and DPO is as follows: Figure 7 As shown, the training process for KTO (Knowledge Transfer Optimization) and DPO (Direct Preference Optimization) includes the following core components: DPO (previous) or KTO (previous): Represents the initial baseline model of the DPO or KTO algorithm, serving as a reference for the legal big language model before optimization, and used to compare the performance differences of the models before and after optimization.

[0086] Phase One: D1→LM policy: In the first phase, the initial legal language model strategy (LMpolicy) is used to generate legal text samples, conduct preliminary legal content generation and evaluation, and provide basic data for subsequent optimization.

[0087] Phase Two: D2 → LM policy: The second phase is the core optimization phase. It uses the initial legal language model strategy (LM policy) to generate legal text samples, conduct preliminary legal content generation and evaluation, and provide basic data for subsequent optimization. Phase Three: D3→LM policy: In the third phase, the optimized Legal Language Model (LMpolicy) is used for final training and fine-tuning to ensure that the model achieves significant and balanced improvements in key legal dimensions (such as the accuracy of legal citations and the logical coherence of judgments), ultimately obtaining an enhanced target Legal Language Model.

[0088] For DPO: Perform direct preference optimization, which directly optimizes the model by comparing the preferences of samples (without pairwise comparisons between samples), and generates legal texts with multi-objective collaborative optimization.

[0089] For KTO: Perform knowledge transfer optimization, and through knowledge transfer from the reference model, efficiently inject legal expertise into the target model to achieve precise optimization in dimensions such as the accuracy of legal citations and the logical coherence of judgments.

[0090] like Figure 7 As shown, the KTO and DPO algorithms are trained sequentially for each objective enhancement of the large legal model, with the training process divided into sequential steps based on different training enhancement objectives.

[0091] In summary, based on the three common training methods mentioned above, the training of mainstream legal language models in multi-objective scenarios mainly involves the following issues: 1) When using single-objective reinforcement learning algorithms such as DPO / KTO for sequential training, there is a serious waste of time and computing power; 2) In the legal field, there may be inherent conflicts between alignment goals across different dimensions. Mainstream algorithms like DPO, MO-DPO, and MO-RLHF primarily use datasets with good and bad data pairs for training. During training, different preference goals are typically handled independently (e.g., training datasets are separate for enhancing harmfulness and usefulness). This fragmented approach is the root cause of instability in multi-objective human alignment: because large language models essentially learn knowledge from training data, attributes such as harmfulness are subjective, and their qualitative boundaries are often blurred. More complexly, different attributes may overlap or potentially conflict. For example, a piece of harmful data might be highly valuable in terms of usefulness, causing the model to deviate from its harmfulness alignment calibration and then undergo a preference calibration in terms of usefulness. This results in significant fluctuations during multi-objective alignment, and in some cases, even model training failure. 3) Reinforcement learning algorithms are inherently unstable, especially when performing reinforcement learning on multiple targets simultaneously. The convergence and stability of these algorithms are extremely poor, making it difficult to achieve the desired reinforcement effect. 4) Constructing preference data is difficult and consumes a lot of manpower and computing power. Existing methods usually require the collection of a large number of preference data pairs, the data labeling cost is high, and it is difficult to accurately capture complex human preferences. Comparing the merits of two data in a certain aspect is always more difficult than directly judging the merits of one data in a certain aspect.

[0092] See Figure 8 , Figure 8 This is a schematic diagram of the MO-KTO algorithm in the MO-KTO legal model enhancement method of the present invention, as shown below. Figure 8 As shown in the diagram, the MO-KTO (Multi-Objective Knowledge Transfer Optimization) algorithm contains the following key elements: Multi-objective definition: The diagram lists the seven core legal dimensions that the large legal model needs to optimize: Politeness: Whether the model output is polite and appropriate; Harmlessness: The model output indicates whether harm or negative content is avoided. Compliance: Whether the model output complies with legal regulations and industry standards; Data: Whether the model output is based on accurate and complete data; Logic: Whether the model output is logically rigorous and the reasoning is reasonable; Usefulness: Whether the model output has practical value to the user; Correctness: Whether the model output is accurate and error-free; Binary signal label: The 1 or 0 marked after each target indicates the sample's satisfaction status on that target: 1: Indicates that the sample meets the objective (e.g., politeness=1 indicates that politeness is output). 0: Indicates that the sample does not meet the target (e.g., logicality = 0 indicates that the output logic is not coherent). Abbreviations: The diagram uses abbreviations for targets (such as Politén → politeness, Harmle → harmlessness) to simplify the display and facilitate quick identification of targets in the algorithm process.

[0093] Loss function calculation: Core formula shown in the figure:

[0094] In the dataset Expected value on Represents the weight vector transpose, For the first The decision weight of each legal optimization objective. For quantification samples In the The value function of an optimization objective.

[0095] LM policy: Legal language model strategy. Through the model optimized by the above loss function, it can achieve synergistic improvement in core legal dimensions such as politeness, harmlessness, and compliance, and generate high-quality output that conforms to legal professional norms.

[0096] It's important to note that a key insight from Kahneman et al.'s prospect theory in 1979 is that humans exhibit risk aversion when faced with gains, and conversely, risk-seeking behavior when faced with losses. This theory allows the value function to vary relative to a reference point. In the KTO algorithm, the reference point... The strategy model is set. and reference model The KL divergence between them.

[0097]

[0098] in, As a reference point, In the dataset Above input The average calculation, For hyperparameters, KL divergence measures the difference between two probability distributions. For the current policy model, at the input Output generated below The probability distribution, For the reference model, at the input Output generated below The probability distribution.

[0099] Relative to this reference point, the sensitivity to loss far exceeds the sensitivity to gains of the same magnitude. Mathematically, this is expressed as the slope of the loss domain being steeper than that of the gain domain, resulting in the following mathematical expression:

[0100] in, Let be the value function, representing the subjective value assessment of a sample's performance on a specific legal optimization objective in the MO-KTO algorithm. This represents the current model's performance on the target. The reference point represents the average performance benchmark of the reference model on this legal optimization objective. is the loss aversion coefficient, indicating that the model is more sensitive to losses than to gains. and Let be the curvature parameter of the value function. To influence the curvature of the revenue portion, The smaller the value, the more gradual the revenue growth. To influence the curvature of the lost portion, The smaller the value, the more gradual the loss growth; Where 0 < , ≤1, according to the characteristic of "loss aversion" in prospect theory, Compare Closer to 0, at the same time This represents the loss aversion parameter, usually A score greater than 1 indicates that the psychological impact of the loss far outweighs the equivalent level of gain.

[0101] Mathematical expression for the value function of KTO:

[0102]

[0103] in, The value function in the KTO algorithm is used to quantify samples. Subjective value in achieving specific legal optimization goals. It is the Sigmoid smoothing function. The reward value for the KTO algorithm. The reference point represents the average performance benchmark of the reference model on this legal optimization objective. For input Below, output Compliant with legal regulations For input Below, output Violation of laws and regulations For hyperparameters, For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability of.

[0104] Decision weight function using symbols This indicates that in KTO, two hyperparameters are used to represent the weights of priority data and non-priority data, respectively:

[0105] in, For the weight function, The weighting coefficients are for the ideal output. Weighting coefficients for unsatisfactory outputs; The loss function of KTO can ultimately be expressed as:

[0106] in, Let KTO be the loss function. For the current legal language model, As a reference model, For the expected value, This is a weighting function that dynamically assigns weights based on the quality (idealism) of the output y, used to adjust the optimization intensity for different outputs. The value function in the KTO algorithm is used to quantify samples. Subjective value in achieving specific legal optimization goals.

[0107] MO-KTO is an exploration to address the problems existing in mainstream large-scale multi-objective reinforcement learning algorithms. It systematically solves the fragmentation problem of training objectives by introducing a multi-bit binary encoding strategy. This method uses hierarchical binary sequences to explicitly encode different preference attributes, using each independent "bit" to define the boundaries between subjective attributes. This achieves fine-grained classification of various preference combinations. Assume that an n-bit binary signal is used to represent the target's preference or dislike in each dimension for different dimensions of the objective. At the same time, configure a weight vector The weight of each target represents its importance; the multi-bit binary sequence distinguishes different preferences, with each bit representing no preference (0) and preference (1).

[0108] For each target Set a reference point This reference point is based on the entire dataset.

[0109]

[0110] in, As a reference point, In the dataset Above input The average calculation, For hyperparameters, KL divergence measures the difference between two probability distributions. For the current policy model, at the input Output generated below The probability distribution, For the reference model, at the input Output generated below The probability distribution.

[0111] value function The mathematical expression:

[0112]

[0113]

[0114] in, For quantification samples In the The value function of an optimization objective The weighting coefficients for the revenue side. The weighting coefficients for the loss side. It is the Sigmoid smoothing function. For the first The reward value for each optimization objective. For the first The baseline reference value for each optimization objective. For input Below, output Compliant with legal regulations For input Below, output Violation of laws and regulations For hyperparameters, For the first The decision weight of each legal optimization objective. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability of.

[0115] Combining all the above equations, the mathematical expression for the loss function of MO-KTO is:

[0116] in, Let MO-KTO be the total loss function of the algorithm. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability, Summing all optimization objectives, In the dataset Expected value on Represents the weight vector transpose, For the first The decision weight of each legal optimization objective. For quantification samples In the The value function of an optimization objective.

[0117] Step S24: Update the parameters of the legal big language model according to the loss function, obtain training process data, optimize the legal big language model according to the training process data, and obtain the optimized target legal big language model.

[0118] It is understandable that by calculating the gradient information of the MO-KTO loss function, the parameters of the legal big language model can be dynamically updated, training process data can be obtained, the legal big language model can be optimized based on the training process data, the optimized target legal big language model can be obtained, and finally the target legal big language model that achieves significant and balanced optimization on key legal indicators can be obtained.

[0119] This embodiment, through the above-described scheme, determines the labeled multi-bit binary signal data using the multi-bit binary signal labels; inputs the legal big language model, the multi-bit binary signal data, and each optimization objective into the MO-KTO algorithm framework; utilizes the MO-KTO algorithm framework to drive multi-objective reinforcement learning through a binary signal alignment mechanism, and calculates the loss function in real time; updates the parameters of the legal big language model according to the loss function to obtain training process data; optimizes the legal big language model based on the training process data to obtain an optimized target legal big language model. This ensures that the core legal dimensions achieve synergistic improvement during training, effectively avoids performance degradation caused by objective conflicts, and ultimately obtains a target legal big language model that is significantly and balancedly optimized in key legal indicators.

[0120] Furthermore, Figure 9 This is a flowchart illustrating the fourth embodiment of the MO-KTO legal model enhancement method of the present invention, as shown below. Figure 9 As shown, based on the first embodiment, a fourth embodiment of the MO-KTO legal model enhancement method of the present invention is proposed. In this embodiment, step S30 specifically includes the following steps: Step S31: Use a preset legal test set to perform a multi-dimensional capability assessment of the target legal language model, calculate the independent quantitative performance index of each optimization target, and generate the assessment results.

[0121] It should be noted that a systematic, multi-dimensional capability assessment of the target legal language model is conducted through a pre-defined, standardized test set covering core legal scenarios such as the accuracy of legal citations and the logical coherence of judgments (e.g., containing 10,000+ high-quality legal case samples). Quantitative performance indicators (such as legal citation accuracy, judgment logical coherence score, and semantic compliance rate) can be independently calculated for each optimization objective, generating a structured evaluation report containing performance data for each objective. This provides objective data support for subsequent assessments of whether the model has achieved significant and balanced improvements in key legal dimensions (e.g., an increase of ≥15% in legal citation accuracy and ≥10% in judgment logical coherence score), ensuring that the evaluation results accurately reflect the model's enhancement effect.

[0122] Step S32: Plot the Pareto front curve based on the evaluation results, and dynamically visualize the trade-offs and non-dominated solution sets among the optimization objectives based on the Pareto front curve.

[0123] Understandably, performance indicators obtained from multi-dimensional capability assessments (such as accuracy of legal citations and scores on logical coherence of judgments) are used to generate Pareto front curves, dynamically visualizing the trade-offs between various legal optimization objectives (for example, an improvement in the accuracy of legal citations may be accompanied by a slight decrease in logical coherence of judgments). Furthermore, non-dominated solution sets (i.e., models that cannot be further optimized for a single objective without compromising other objectives) are automatically identified. This provides a clear picture of the model's optimization balance in the legal professional dimension, offering an objective and visual basis for determining whether the target legal language model has achieved "significant and balanced" enhancements (e.g., key indicators improve by ≥10% simultaneously and the Pareto front converges to the ideal region).

[0124] Step S33: When the performance improvement of the preset key legal target among the optimization targets is detected, and all indicators of the preset key legal target reach the preset legal professional threshold, it is determined that the target legal language model enhancement is successful.

[0125] It should be understood that when determining the success of the enhancement of the target legal language model, it is permissible to rigorously test whether the performance indicators of the preset key legal objectives (such as the accuracy of legal citation, the logical coherence of judgment, and the semantic compliance rate) have achieved significant improvements (e.g., the accuracy of legal citation is ≥15% higher than the initial model, and the score of logical coherence of judgment is ≥10%), and the quantitative indicators of all key objectives simultaneously reach the preset legal professional thresholds (e.g., the accuracy of legal citation is ≥85%, and the score of logical coherence of judgment is ≥80%). At the same time, Pareto frontier analysis is used to ensure that there is no degradation among the objectives (e.g., the fairness indicator does not decrease due to the optimization of other objectives). This accurately confirms that the model has achieved balanced and practical professional enhancement in the core legal dimensions, avoiding the problem of overall performance imbalance caused by the over-optimization of a single objective in traditional methods.

[0126] It should be noted that improving the reinforcement learning algorithm for large legal models in multi-objective scenarios based on the MO-KTO algorithm principle has the following advantages: 1) Effectively alleviates multi-objective conflicts and improves overall model performance: Existing multi-objective reinforcement methods for large-scale legal models often fail to achieve ideal results when dealing with multi-dimensional alignment objectives due to inherent conflicts between objectives. This invention assigns independent binary signal bits to each objective, enabling the model to independently perceive and learn the preferences of each objective during reinforcement learning. This mechanism effectively decouples the complex relationships between different objectives, avoiding performance degradation or suboptimal solutions caused by objective conflicts in traditional methods. Ultimately, the model can achieve a better balance and improved overall performance across multiple interrelated and potentially conflicting dimensions of legal knowledge, generating more comprehensive legal texts that meet diverse requirements.

[0127] 2) Significantly Improves the Stability and Efficiency of Reinforcement Learning: Traditional multi-objective reinforcement learning methods, especially when the number of objectives increases, often face problems such as unstable training and difficulty in convergence. This invention abandons complex reward model construction and weight allocation mechanisms, directly utilizing simple and efficient multi-bit binary signals as alignment criteria, combined with homogeneous preference alignment algorithms such as KTO for training. This direct and fine-grained signal feedback mechanism enables the model to learn and optimize more stably, accelerates convergence, and reduces the risk of oscillations or divergence during training. Therefore, this invention can complete multi-objective reinforcement tasks for large legal models with higher efficiency and stability.

[0128] 3) Reduce data annotation costs and simplify data preparation: Existing methods typically require collecting a large amount of preference pair data (i.e., comparing which of two samples is better) and training a separate reward model for each preference. This not only requires significant manpower and resources for high-quality data annotation, but the annotation process is also complex and error-prone. This invention uses multi-bit binary signals, where each bit only needs to determine whether the data sample conforms to the preference (0 or 1) for a specific objective. This binary annotation method greatly simplifies the annotation difficulty and workload. By reducing the complexity and cost of data annotation, this invention makes the multi-objective enhancement technology for large-scale legal models more operable and has greater potential for widespread application.

[0129] 4) Refined characterization of data alignment attributes to improve model learning accuracy: Traditional methods often simply categorize data as "good" or "bad," failing to capture subtle differences and complex attributes across different dimensions. The multi-bit binary signal introduced in this invention can more precisely and comprehensively characterize the alignment attributes of data samples across multiple objectives in the form of "multi-level signals." For example, a legal text might excel in "accuracy" but lack in "conciseness," a distinction that traditional methods struggle to make effectively. However, multi-bit binary signals can clearly express this complex information. This refined characterization of alignment attributes allows the model to more accurately understand and learn human preferences, thereby generating higher-quality legal texts that better meet specific needs.

[0130] 5) Enhancing the model's adaptability to complex legal scenarios: Legal knowledge is complex and ever-changing, and different cases or situations may have subtle differences in the requirements for legal texts. This invention uses multi-bit binary signals to perform fine-grained control over multiple objectives, enabling the model to better adapt to this complexity. By adjusting or combining different binary signals, the model can be flexibly guided to generate texts that meet different preference combinations under specific legal scenarios, thereby significantly enhancing the flexibility and adaptability of the large-scale legal model in practical applications.

[0131] In the specific implementation, see Figure 10 , Figure 10 For the MO-KTO legal enhancement system architecture diagram, such as Figure 10 As shown, the legal enhancement system architecture based on the MO-KTO algorithm mainly includes five modules: a legal big language model module, a multi-bit binary signal generation and annotation module, a data management module, a multi-objective reinforcement learning module, and an evaluation and supervision module.

[0132] The legal language model module serves as the foundational model, primarily responsible for generating text content and updating parameters during subsequent reinforcement learning. The multi-bit binary signal generation and annotation module defines multiple objectives and generates corresponding multi-bit binary signals based on the performance of legal text samples on these objectives. The data management module stores and manages raw legal text data, annotated multi-bit binary signal data, data generated during model training, and final evaluation data. The multi-objective reinforcement learning module is the core of the system, utilizing multi-bit binary signals and the MO-KTO algorithm for reinforcement learning on the large legal model. The evaluation and supervision module assesses the model's capabilities for each objective, visualizing the multi-objective evaluation by plotting the Pareto front.

[0133] Step 1: Deploy the legal big language model module, load the pre-trained legal big model as the base model, and complete the initial configuration to ensure that it has the ability to generate text and update parameters.

[0134] Step 2: Construct a multi-bit binary signal generation and annotation module, define multiple law-related optimization objectives (such as the accuracy of legal citation, the logical coherence of judgments, semantic compliance, fairness, etc.), and generate corresponding multi-bit binary signal labels (each bit represents whether an objective is satisfied) based on the performance of the original legal text sample on each objective.

[0135] Step 3: Import and store the original legal text dataset through the data management module, and simultaneously save the multi-bit binary signal annotation results generated in Step 2 to establish a structured data warehouse to support data retrieval and version management for subsequent reinforcement learning.

[0136] Step 4: Start the multi-objective reinforcement learning module, input the legal large language model, the labeled multi-bit binary signal data and the set of optimization objectives into the MO-KTO algorithm framework, perform parameter fine-tuning and policy update, so that the model can be optimized collaboratively on multiple legal objectives.

[0137] Step 5: During the training process, the intermediate models, loss values, gradient information and other outputs generated by the MO-KTO algorithm are continuously sent back to the data management module to realize full recording and traceable management of training process data.

[0138] Step 6: Activate the evaluation and supervision module, use an independent legal test set to evaluate the current enhanced model in multiple dimensions, calculate the performance indicators (such as accuracy, F1 score, logical consistency score, etc.) under each objective, and generate an evaluation report.

[0139] Step 7: Based on the evaluation results, plot the Pareto front curve in the evaluation and supervision module to visualize the trade-offs between different objectives, identify the non-dominated solution set, and help determine whether the model has reached a multi-objective equilibrium state.

[0140] Step 8: By comparing the model performance at different iteration stages on the Pareto front, analyze the improvement effect of the MO-KTO algorithm on each objective of the large legal model. If the expected results are not met, return to step 4 to continue iterative optimization.

[0141] Step 9: When the evaluation results show that the model has achieved significant and balanced improvements in key legal objectives and can stably output text that conforms to judicial logic, the legal model enhancement is confirmed to be successful, and the process ends.

[0142] This embodiment, through the above-described scheme, uses a preset legal test set to perform multi-dimensional capability evaluation on the target legal language model, calculates the independent quantitative performance indicators of each optimization objective, and generates evaluation results; based on the evaluation results, it plots Pareto front curves and dynamically visualizes the trade-offs and non-dominated solution sets among the optimization objectives; when the performance improvement of the preset key legal objectives among the optimization objectives is detected, and all indicators of the preset key legal objectives reach the preset legal professional threshold, the target legal language model is deemed to have been successfully enhanced; this effectively avoids the performance degradation caused by objective conflict in traditional multi-objective reinforcement learning; ensures significant and balanced improvement in key legal dimensions, thereby accurately verifying the successful enhancement of the target legal language model, effectively alleviating multi-objective conflict, improving the overall performance of the model, significantly improving the stability and efficiency of reinforcement learning, reducing data annotation costs, simplifying the data preparation process, finely characterizing data alignment attributes, improving model learning accuracy, and enhancing the model's adaptability to complex legal scenarios.

[0143] Accordingly, the present invention further provides a MO-KTO legal model enhancement device.

[0144] Reference Figure 11 , Figure 11 This is a functional block diagram of the first embodiment of the MO-KTO legal model enhancement device of the present invention.

[0145] In a first embodiment of the MO-KTO legal model enhancement device of the present invention, the MO-KTO legal model enhancement device includes: The tag generation module 10 is used to obtain several legally related optimization targets and generate multi-bit binary signal tags based on the performance of the original legal text sample in each optimization target.

[0146] The model optimization module 20 is used to input the legal big language model, the multi-bit binary signal data labeled with the multi-bit binary signal labels and each optimization target into the MO-KTO algorithm framework to obtain training process data, and optimize the legal big language model based on the training process data to obtain the optimized target legal big language model.

[0147] The judgment module 30 is used to perform multi-dimensional capability evaluation on the target legal language model using a preset legal test set, obtain performance indicators under each optimization objective, and judge whether the target legal language model has been successfully enhanced based on the performance indicators.

[0148] The steps for implementing each functional module of the MO-KTO legal model enhancement device can be referred to in the various embodiments of the MO-KTO legal model enhancement method of the present invention, and will not be repeated here.

[0149] Furthermore, this embodiment of the invention also proposes a storage medium storing a MO-KTO legal model enhancement program, which, when executed by a processor, implements the operations described in the above embodiments of the MO-KTO legal model enhancement method.

[0150] Those skilled in the art will understand that all or part of the steps in the methods described above can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium is a computer-readable storage medium, including: a USB flash drive, a portable hard drive, and a read-only memory (ROM). Various media that can store program code, such as only memory, random access memory (RAM), magnetic disks or optical disks.

[0151] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0152] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0153] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for enhancing the MO-KTO legal model, characterized in that, The MO-KTO legal model enhancement method includes: Obtain several law-related optimization objectives, and generate multi-bit binary signal labels based on the performance of the original legal text samples on each optimization objective; The legal big language model is input into the MO-KTO algorithm framework based on the multi-bit binary signal data labeled with the multi-bit binary signal tags and each optimization target to obtain training process data. The legal big language model is then optimized based on the training process data to obtain the optimized target legal big language model. The target legal language model is evaluated in multiple dimensions using a preset legal test set to obtain performance indicators under each optimization objective. Based on the performance indicators, it is determined whether the target legal language model has been successfully enhanced.

2. The MO-KTO legal model enhancement method as described in claim 1, characterized in that, The process involves acquiring several law-related optimization objectives, and generating multi-bit binary signal tags based on the performance of the original legal text samples on each optimization objective, including: The optimization goals are to obtain the accuracy of the corresponding legal provisions, the logical coherence of the judgment, the semantic compliance, and the fairness. Binary signal labels for each optimization objective are generated based on the actual performance of the original legal text sample on each optimization objective.

3. The MO-KTO legal model enhancement method as described in claim 1, characterized in that, The process of inputting the legal big language model, based on the multi-bit binary signal data labeled with the multi-bit binary signal tags and various optimization objectives, into the MO-KTO algorithm framework to obtain training process data, and optimizing the legal big language model based on the training process data to obtain the optimized target legal big language model, includes: The labeled multi-bit binary signal data is determined based on the multi-bit binary signal label; The legal language model, the multi-bit binary signal data, and each optimization objective are input into the MO-KTO algorithm framework; The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism, and the loss function is calculated in real time. The parameters of the legal big language model are updated according to the loss function to obtain training process data. The legal big language model is then optimized based on the training process data to obtain the optimized target legal big language model.

4. The MO-KTO legal model enhancement method as described in claim 3, characterized in that, The method of using the MO-KTO algorithm framework to drive multi-objective reinforcement learning through a binary signal alignment mechanism and to calculate the loss function in real time includes: The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism to obtain the value function and the decision weights of each optimization objective; A loss function is dynamically constructed based on the value function and the decision weights.

5. The MO-KTO legal model enhancement method as described in claim 4, characterized in that, The process of using the MO-KTO algorithm framework to drive multi-objective reinforcement learning through a binary signal alignment mechanism to obtain the value function and the decision weights of each optimization objective includes: The MO-KTO algorithm framework is used to drive multi-objective reinforcement learning through a binary signal alignment mechanism. The value function and decision weights for each optimization objective are obtained through the following formula: in, For quantification samples In the The value function of an optimization objective The weighting coefficients for the revenue side. The weighting coefficients for the loss side. It is the Sigmoid smoothing function. For the first The reward value for each optimization objective. For the first The baseline reference value for each optimization objective. For input Below, output Compliant with legal regulations For input Below, output Violation of laws and regulations For hyperparameters, For the first The decision weight of each legal optimization objective. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability of.

6. The MO-KTO legal model enhancement method as described in claim 4, characterized in that, The step of dynamically constructing a loss function based on the value function and the decision weights includes: Based on the value function and the decision weights, a loss function is dynamically constructed using the following formula: in, Let MO-KTO be the total loss function of the algorithm. For the current policy model at input Output generated below The probability, As a reference model in the input Output generated below The probability, Summing all optimization objectives, In the dataset Expected value on Represents the weight vector transpose, For the first The decision weight of each legal optimization objective. For quantification samples In the The value function of an optimization objective.

7. The MO-KTO legal model enhancement method as described in claim 1, characterized in that, The process of using a preset legal test set to perform a multi-dimensional capability evaluation of the target legal language model, obtaining performance indicators under each optimization objective, and determining whether the target legal language model has been successfully enhanced based on the performance indicators includes: The target legal language model is evaluated in multiple dimensions using a preset legal test set. Independent quantitative performance indicators for each optimization objective are calculated, and evaluation results are generated. Based on the evaluation results, Pareto front curves are plotted, and the trade-offs and non-dominated solution sets among the optimization objectives are dynamically visualized based on the Pareto front curves. When the performance improvement of the preset key legal objectives among the optimization objectives is detected, and all indicators of the preset key legal objectives reach the preset legal professional threshold, the target legal language model is determined to be successfully enhanced.

8. A MO-KTO legal model enhancement device, characterized in that, The MO-KTO legal model enhancement device includes: The tag generation module is used to obtain several legally related optimization objectives and generate multi-bit binary signal tags based on the performance of the original legal text sample on each optimization objective. The model optimization module is used to input the legal big language model, the multi-bit binary signal data labeled with the multi-bit binary signal labels and each optimization target into the MO-KTO algorithm framework to obtain training process data, and optimize the legal big language model based on the training process data to obtain the optimized target legal big language model. The judgment module is used to perform multi-dimensional capability evaluation on the target legal language model using a preset legal test set, obtain performance indicators under each optimization objective, and judge whether the target legal language model has been successfully enhanced based on the performance indicators.

9. A MO-KTO legal model enhancement device, characterized in that, The MO-KTO legal model enhancement device includes: a memory, a processor, and a MO-KTO legal model enhancement program stored on the memory and executable on the processor, the MO-KTO legal model enhancement program being configured to implement the steps of the MO-KTO legal model enhancement method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a MO-KTO legal model enhancement program, which, when executed by a processor, implements the steps of the MO-KTO legal model enhancement method as described in any one of claims 1 to 7.

Images