Judicial big model effect evaluation method based on multiple dimensions and program product

Abstract

The invention relates to the technical field of software, in particular to a judicial big model effect evaluation method based on multiple dimensions and a program product. Comprising the following steps: obtaining a generation result of a large model and a corresponding reference answer; and calculating the similarity between the generated result and the reference answer from a word dimension, a statement dimension, reinforcement learning and a human feedback dimension, then carrying out normalization operation on the three numerical values, and then carrying out weighted summation to obtain a final evaluation score. According to the method, the generation effect of the large model is evaluated from multiple dimensions, and by using the evaluation, a better training effect can be obtained in the large model training process.

Description

Technical Field

[0001] The present invention relates to the field of software technology, and in particular to a multi-dimensional judicial big model effect evaluation method and program product. Background Art

[0002] The current era is artificial intelligence and big models. Major research institutions and companies at home and abroad pay great attention to the development and application of big model technology. Similarly, the judicial field also needs to train its own judicial big models to cope with the development of the artificial intelligence era and changes in customer needs. Big models mainly use generative artificial intelligence methods. The generation effect of the model requires a scientific evaluation method, otherwise the training process of the model will be ineffective. Therefore, the construction of an effect evaluation system is crucial. The commonly used text generation evaluation methods are mainly based on BLEU and ROUGE based on word overlap rates. Of course, this method also has its limitations. For example, it may cause it to pay too much attention to the surface form of words rather than their deep meaning or semantics, ignore the order of words, lack of evaluation of grammar and semantics, etc. In particular, the general method does not consider the judicial context and legal professional vocabulary. Therefore, for the generation effect of judicial big models, a more mature evaluation system needs to be built. Summary of the invention

[0003] In view of the above problems, the present invention proposes a technical solution. The present invention is a multi-dimensional judicial large model effect evaluation method. The method of the present invention can well solve the evaluation problem of the large model generation effect, which is conducive to evaluating the training effect of the model. It can be used as an effect evaluation standard, which is conducive to adjusting the batch size, learning rate, lora_rank and other hyperparameters, as well as the data type, scale, training rounds, duration and other variables when training the large model, to obtain an optimal solution and determine the parameter selection.

[0004] The present invention provides a multi-dimensional judicial big model effect evaluation method, comprising the following steps:

[0005] S1: Obtain the generation result of the large model and the corresponding reference answer, where the generation result and the reference answer are both in the form of strings;

[0006] Perform word segmentation operation on the generated result and the reference answer to obtain the generated result after word segmentation and the reference answer after word segmentation;

[0007] S2: Calculate the similarity between the generated result and the reference answer from the word dimension:

[0008]

[0009] in, Indicates the similarity between the generated result obtained from the word dimension and the reference answer;

[0010] The larger the value, the more similar it is;

[0011] Indicates the Jaccard coefficient between the generated result after word segmentation and the reference answer after word segmentation;

[0012] Represents the cosine similarity between the generated result after word segmentation and the reference answer after word segmentation;

[0013] Represents the Manhattan distance between the generated result after word segmentation and the reference answer after word segmentation;

[0014] Represents the edit distance between the generated result after word segmentation and the reference answer after word segmentation;

[0015] S3: Calculate the similarity between the generated result and the reference answer from the sentence dimension:

[0016]

[0017] in, Indicates the similarity between the generated result obtained from the sentence dimension and the reference answer;

[0018] The larger the value, the more similar it is;

[0019] represents the common model similarity;

[0020] Indicates the legal model similarity;

[0021] in, The calculation method is:

[0022] The generated results and the reference answers are converted into vectors through the embedding layer of the general natural language processing model, and then the similarity between the generated results in vector form and the reference answers is calculated to obtain ;

[0023] The calculation method is:

[0024] The generated results and the reference answers are converted into vectors through the embedding layer of the natural language processing model trained with legal data, and then the similarity between the generated results in vector form and the reference answers is calculated to obtain ;

[0025] S4: Reinforcement learning and human feedback dimension calculates the similarity between the generated results and the reference answer:

[0026] Input the generated results and the reference answers into the reward model to obtain a similarity score ;

[0027] The reward model is a deep learning model, which is obtained by pre-training with manually labeled data;

[0028] Finally, we get:

[0029]

[0030] in, Indicates the similarity between the generated results obtained from reinforcement learning and human feedback dimensions and the reference answer;

[0031] The larger the value, the more similar it is;

[0032] S5: Yes 1. , The numerical value is normalized;

[0033] Calculate the final score between the generated result and the reference answer:

[0034]

[0035] in, Indicates the final evaluation score between the generated result and the reference answer;

[0036] The larger the value, the more similar it is;

[0037] , , They are , , The weight of .

[0038] Preferably, , , The values ​​are 0.2, 0.3, and 0.5 respectively.

[0039] Preferably, the word segmentation method of the word segmentation operation is vocabulary-based word segmentation.

[0040] Preferably, when calculating the cosine similarity and Manhattan distance between the generated result after word segmentation and the reference answer after word segmentation, the generated result after word segmentation and the reference answer after word segmentation are both converted into vectors through one-hot encoding and then calculated.

[0041] Preferably, the natural language processing model is Bert.

[0042] Preferably, when calculating the similarity between the generated result in vector form and the reference answer, cosine similarity is used as an indicator for evaluating the similarity.

[0043] The present invention also provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0044] Beneficial effect: The present invention evaluates the generation effect of the large model from multiple dimensions. By using this evaluation, it is beneficial to obtain better training effects during the large model training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the present invention; DETAILED DESCRIPTION

[0046] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Because the present invention is based on a multi-dimensional evaluation method, different processing needs to be done on the input according to the difference in each dimension. For example, in the word base dimension, word segmentation needs to be done on the input before subsequent similarity and distance calculations can be performed. The other two dimensions are calculated based on the deep learning model, so only the original string content needs to be input, and then the embedding layer of the model will convert the original content into an embedded vector for the remaining calculations.

[0048] like Figure 1 As shown, a multi-dimensional judicial big model effect evaluation method includes the following steps:

[0049] S1: Get the generation result of the large model and the corresponding reference answer, where the generation result and the reference answer are both in the form of strings;

[0050] Perform word segmentation operation on the generated result and the reference answer to obtain the generated result after word segmentation and the reference answer after word segmentation;

[0051] In the present invention, the basic technical effects can be achieved by using the usual word segmentation method. For example, the generated result or reference answer is "Glory of the King", and the generated result or reference answer after word segmentation is ["King", "Glory"];

[0052] As a preferred implementation, the word segmentation method is word segmentation based on a vocabulary;

[0053] S2: Calculate the similarity between the generated result and the reference answer from the word base dimension:

[0054]

[0055] in, Indicates the similarity between the generated result obtained from the word dimension and the reference answer;

[0056] The larger the value, the more similar it is;

[0057] Indicates the Jaccard coefficient between the generated result after word segmentation and the reference answer after word segmentation;

[0058] Represents the cosine similarity between the generated result after word segmentation and the reference answer after word segmentation;

[0059] Represents the Manhattan distance between the generated result after word segmentation and the reference answer after word segmentation;

[0060] Represents the edit distance (also called Levenshtein distance) between the generated result after word segmentation and the reference answer after word segmentation;

[0061] Specifically, when calculating the cosine similarity and Manhattan distance between the generated result after word segmentation and the reference answer after word segmentation, the generated result after word segmentation and the reference answer after word segmentation are converted into vectors through one-hot encoding and then calculated; in this process, using other encoding methods to convert into vectors can also achieve basic technical effects

[0062] S3: Calculate the similarity between the generated result and the reference answer from the sentence base:

[0063]

[0064] in, Indicates the similarity between the generated result obtained from the sentence dimension and the reference answer;

[0065] The larger the value, the more similar it is;

[0066] represents the common model similarity;

[0067] Indicates the legal model similarity;

[0068] in, The calculation method is:

[0069] The generated results and the reference answers are converted into vectors through the embedding layer of the general natural language processing model, and then the similarity between the generated results in vector form and the reference answers is calculated to obtain ;

[0070] The calculation method is:

[0071] The generated results and the reference answers are converted into vectors through the embedding layer of the natural language processing model trained with legal data, and then the similarity between the generated results in vector form and the reference answers is calculated to obtain ;

[0072] In this embodiment, the natural language processing model is Bert; the natural language processing model trained with legal data is obtained by training Bert with legal data, which is referred to as ;

[0073] Calculate the similarity between the generated result in vector form and the reference answer. Usually cosine similarity is used. Other methods can also achieve basic technical effects.

[0074] S4: The reinforcement learning from human feedback (RLHF) dimension (LawRLHF) calculates the similarity between the generated results and the reference answer:

[0075] Input the generated results and the reference answers into the reward model to obtain the similarity score ;

[0076] The reward model is a deep learning model, which is obtained by pre-training with manually labeled data;

[0077] Finally, we get:

[0078]

[0079] in, Indicates the similarity between the generated results obtained from reinforcement learning and human feedback dimensions and the reference answer;

[0080] The larger the value, the more similar it is;

[0081] S5: Yes 1. , The numerical value is normalized;

[0082] Calculate the final score between the generated result and the reference answer:

[0083]

[0084] in, Indicates the final evaluation score between the generated result and the reference answer;

[0085] The larger the value, the more similar it is;

[0086] , , They are , , The weight of

[0087] In this embodiment, it is found through experiments that when , , When the values ​​of are 0.2, 0.3, and 0.5 respectively, the effect is better.

[0088] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-dimensional judicial big model effect evaluation method, characterized in that: The following steps are involved: S1: Obtain the generation result of the large model and the corresponding reference answer, where the generation result and the reference answer are both in the form of strings; Perform word segmentation operation on the generated result and the reference answer to obtain the generated result after word segmentation and the reference answer after word segmentation; S2: Calculate the similarity between the generated result and the reference answer from the word dimension: in, Indicates the similarity between the generated result obtained from the word dimension and the reference answer; The larger the value, the more similar it is; Indicates the Jaccard coefficient between the generated result after word segmentation and the reference answer after word segmentation; Represents the cosine similarity between the generated result after word segmentation and the reference answer after word segmentation; Represents the Manhattan distance between the generated result after word segmentation and the reference answer after word segmentation; Represents the edit distance between the generated result after word segmentation and the reference answer after word segmentation; S3: Calculate the similarity between the generated result and the reference answer from the sentence dimension: in, Indicates the similarity between the generated result obtained from the sentence dimension and the reference answer; The larger the value, the more similar it is; represents the common model similarity; Indicates the legal model similarity; in, The calculation method is: The generated results and the reference answers are converted into vectors through the embedding layer of the general natural language processing model, and then the similarity between the generated results in vector form and the reference answers is calculated to obtain ; The calculation method is: The generated results and the reference answers are converted into vectors through the embedding layer of the natural language processing model trained with legal data, and then the similarity between the generated results in vector form and the reference answers is calculated to obtain ; S4: Reinforcement learning and human feedback dimension calculates the similarity between the generated results and the reference answer: Input the generated results and the reference answers into the reward model to obtain a similarity score ; The reward model is a deep learning model, which is obtained by pre-training with manually labeled data; Finally, we get: in, Indicates the similarity between the generated results obtained from reinforcement learning and human feedback dimensions and the reference answer; The larger the value, the more similar it is; S5: Yes 1. , The numerical value is normalized; Calculate the final score between the generated result and the reference answer: in, Indicates the final evaluation score between the generated result and the reference answer; The larger the value, the more similar it is; , , They are , , The weight of .

2. According to the multi-dimensional judicial big model effect evaluation method of claim 1, it is characterized in that: , , The values ​​are 0.2, 0.3, and 0.5 respectively.

3. According to the multi-dimensional judicial big model effect evaluation method of claim 1, it is characterized in that: The word segmentation method of the word segmentation operation is word segmentation based on a vocabulary table.

4. According to the multi-dimensional judicial big model effect evaluation method of claim 1, it is characterized in that: When calculating the cosine similarity and Manhattan distance between the generated result after word segmentation and the reference answer after word segmentation, the generated result after word segmentation and the reference answer after word segmentation are converted into vectors through one-hot encoding before calculation.

5. According to the multi-dimensional judicial big model effect evaluation method of claim 1, it is characterized in that: The natural language processing model is Bert.

6. The multi-dimensional judicial big model effect evaluation method according to claim 1 is characterized in that: When calculating the similarity between the generated result in vector form and the reference answer, cosine similarity is used as an indicator to evaluate the similarity.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images