Electronic data judicial expertise suggestion quality evaluation method and system based on Lma model
Through the quality evaluation method of electronic data judicial appraisal opinion based on the Llama model, combined with training data construction, low-order adaptive algorithm fine-tuning and retrieval enhancement generation technology, the problem of the lack of professionalism and timeliness in the quality evaluation of the judicial appraisal field is solved, and efficient and accurate quality evaluation results are achieved.
Patent Information
- Application Number
- CN202411979797.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-06
AI Technical Summary
When applied to the quality evaluation of electronic data judicial appraisal opinions, the existing technology cannot understand the content in the field of judicial appraisal, lacks professionalism and targetedness, and has poor timeliness of knowledge update, which cannot meet the quality evaluation needs of judicial appraisal.
The quality evaluation method of electronic data judicial appraisal opinion based on the Llama model is adopted, and the model is improved in the field of judicial appraisal by constructing training data, supervision and fine-tuning training of low-order adaptive algorithms, search enhancement generation technology and prompt engineering.
It improves the efficiency and accuracy of the quality evaluation of the electronic data judicial appraisal opinion, enhances the timeliness and flexibility of the evaluation method, reduces the cost of knowledge updates, and meets the quality evaluation needs of judicial appraisal.
Smart Images
Figure CN119938912A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of judicial identification, and in particular to a method and system for evaluating the quality of electronic data judicial identification opinions based on a Llama model. Background Art
[0002] Electronic data forensic appraisal refers to the use of computer science theories and technologies, as well as other relevant expertise and experience, to judge, identify and issue appraisal opinions on the relevant content of electronic data involved in litigation. The results of forensic appraisals are related to the realization of judicial justice and the protection of the legitimate rights and interests of the people. Therefore, it is necessary to strictly evaluate and control the process and results of forensic appraisals to ensure the reliability of forensic appraisal opinions. Among them, whether the appraisal procedures comply with the relevant laws and regulations, and whether the appraisal process and appraisal opinions comply with national standards, industry standards, and technical specifications are key steps in the quality assessment of forensic appraisal opinions.
[0003] The traditional method of evaluating the quality of judicial appraisal opinions mainly relies on manual operation. With the rapid development of communication technology, the Internet and other scientific and technological technologies, the demand for electronic data judicial appraisal is growing, and the number of cases is increasing year by year. This traditional manual evaluation method is inefficient and can no longer meet the growing demand for quality evaluation.
[0004] In recent years, in view of the in-depth application of big models in various industries and the remarkable results achieved, some existing technologies have applied big models to the quality assessment of texts such as dialogues, experimental reports, and translations. For example, each experimental requirement is automatically parsed and converted into an evaluation decision tree, the corresponding evaluation decision tree is retrieved according to the experimental task, and each experimental topic is rated using the evaluation decision tree. Combined with the designed prompt word template and the scoring results of each topic, the big model is called to generate the final experimental comments. In addition, the dialogue question and answer data to be evaluated is divided into an objective question and answer data group and a subjective question and answer data group, and evaluation instructions are constructed to obtain the first answer quality evaluation value and the second answer quality evaluation value of the model to be evaluated. After comprehensive judgment, the evaluation grade result based on the big model is obtained.
[0005] However, these existing technologies have many deficiencies when applied to the quality assessment of electronic data judicial appraisal opinions. First, the existing technologies cannot understand the content of electronic data judicial appraisal opinions and are not targeted. The existing technologies have not been trained in the vertical field of electronic data judicial appraisal, so the existing technologies lack knowledge in the field of electronic book judicial appraisal, and the accuracy of directly applying the existing technologies to the quality assessment of electronic data judicial appraisal opinions is low. Second, the existing technologies cannot meet the requirements of judicial appraisal quality assessment and are not professional. Judicial appraisal requires strict compliance with and adoption of the standards, specifications and methods of the professional field. The quality assessment of electronic data judicial appraisal has special requirements, and it is necessary to evaluate the operation sequence, integrity and logic of the document reflected in the text content of the appraisal opinion. The existing technologies cannot be directly applied to the quality assessment of judicial appraisal opinions. Third, the knowledge update timeliness of the existing technologies is poor, and there is a lack of professional knowledge in the field of judicial appraisal. The judicial appraisal standards and judicial appraisal case scoring standards are updated and iterated quickly, so the evaluation method needs to have a flexible update mechanism. Summary of the invention
[0006] In order to solve the above technical problems, the present invention provides a method for evaluating the quality of electronic data forensic appraisal opinions based on the Llama model; on the other hand, it also provides a system for evaluating the quality of electronic data forensic appraisal opinions based on the Llama model.
[0007] The technical problem solved by the present invention can be achieved by adopting the following technical solutions:
[0008] The first aspect of the present invention is to provide a method for evaluating the quality of electronic data forensic expert opinion based on the Llama model, comprising:
[0009] Step S1, constructing training data based on the quality assessment data of the electronic data forensic appraisal opinion;
[0010] Step S2, based on the training data, using a low-order adaptive algorithm to perform supervised fine-tuning training on the large language model to obtain a quality assessment model applied to the vertical field of quality assessment of judicial appraisal opinions;
[0011] Step S3, judging whether the appraisal standards used in the judicial appraisal opinion to be evaluated are applicable and correct;
[0012] If yes, proceed to step S4;
[0013] If not, the judicial appraisal opinion to be evaluated is judged to be unqualified and the reasons are output;
[0014] Step S4, based on the retrieval enhancement generation model and the prompting project, the quality assessment model outputs a quality assessment result for the judicial appraisal opinion to be assessed.
[0015] As a preferred embodiment, the step S1 includes:
[0016] Step S11, filtering the quality assessment data to obtain first screening data;
[0017] Step S12: desensitize the first screening data to obtain second screening data, and use the second screening data as the training data.
[0018] As a preferred implementation, in step S11, the quality assessment data is extracted based on preset filtering rules to obtain the first screening data, which at least includes any one or more combinations of identification text content, expert evaluation scores and expert evaluation opinions.
[0019] As a preferred implementation, in step S12, the sensitive information in the identification text is replaced with preset characters to obtain the second screening data after data desensitization.
[0020] As a preferred implementation, the large language model is a Llama model.
[0021] As a preferred embodiment, the step S2 includes:
[0022] Step S21, segmenting and encoding the training data to obtain an identification opinion vector;
[0023] Step S22, freezing the first weight parameter of the large language model;
[0024] Step S23, adding a trainable second weight parameter to the cross attention layer of the large language model;
[0025] Step S24, training or updating the second weight parameter according to the identification opinion vector to obtain the trained or updated second weight parameter;
[0026] Step S25, merging the trained or updated second weight parameter with the frozen first weight parameter to obtain a merged weight parameter, and constructing the quality assessment model based on the merged weight parameter.
[0027] As a preferred embodiment, the step S3 includes:
[0028] Step S31, extracting the entrusted matters and the appraisal standards used in the judicial appraisal opinion to be evaluated based on the quality assessment model;
[0029] Step S32, based on the pre-constructed electronic data judicial appraisal standard applicable rule library, determine whether the appraisal standard used in the judicial appraisal opinion to be evaluated is correctly applied and whether it meets the requirements of the entrusted matter.
[0030] As a preferred embodiment, the step S4 includes:
[0031] Step S41, based on the identification standard terms used in the judicial identification opinion to be evaluated, calling the retrieval enhancement generation model to obtain the identification standard related knowledge content, and using the quality assessment model to summarize the identification steps based on the identification standard related knowledge content and the first prompt word;
[0032] Step S42, inputting the judicial appraisal opinion to be evaluated and the appraisal steps summarized into the prompt word template, using the quality evaluation model to make a judgment, and outputting the judgment results of completeness, sequence and logic;
[0033] Step S43, calculating the evaluation score according to the deduction reasons and the corresponding deduction scores in the integrity, sequentiality and logic judgment results, and then inputting the deduction reasons into the quality assessment model to generate the quality assessment result.
[0034] As a preferred implementation, the step S42 includes:
[0035] Step S421, using the quality assessment model, outputting the integrity judgment result according to the second prompt word, wherein the task description of the second prompt word includes the full text of the forensic expert opinion to be evaluated, the summarized appraisal steps, and instructions for judging whether the appraisal process complies with the inspection steps required by the appraisal standard, and the output format includes the step number, answer, reason and the corresponding original content of the appraisal opinion;
[0036] Step S422, comparing the positions of the original contents of the appraisal opinions corresponding to the serial numbers of two adjacent steps in the judicial appraisal opinion to be evaluated with the operation sequence of the appraisal standards, and outputting a sequential judgment result;
[0037] Step S423, using the quality assessment model, output a logical judgment result based on the third prompt word. The task description of the third prompt word includes the full text of the judicial appraisal opinion to be evaluated and the quality of judging whether the appraisal process is consistent with the appraisal opinion and whether the appraisal opinion corresponds one-to-one with the entrusted matters. The output format includes the reason for deduction and the deduction score.
[0038] The second aspect of the present invention is to provide a quality assessment system for electronic data forensic expert opinion based on the Llama model, which is used to implement the quality assessment method for electronic data forensic expert opinion based on the Llama model as described above, including:
[0039] A training data construction module, used to construct training data based on quality assessment data of electronic data forensic appraisal opinions;
[0040] A model training module, connected to the training data building module, is used to perform supervised fine-tuning training on a large language model using a low-order adaptive algorithm based on the training data to obtain a quality assessment model applied to the vertical field of quality assessment of judicial appraisal opinions;
[0041] The quality assessment module is connected to the model training module and is used to determine whether the appraisal standards used in the judicial appraisal opinions to be evaluated are correctly applied; if they are wrong, the judicial appraisal opinions to be evaluated are judged to be unqualified and the reasons are output; when the appraisal standards are correctly applied, the quality assessment model outputs the quality assessment results for the judicial appraisal opinions to be evaluated based on the retrieval enhancement generation model and prompt engineering.
[0042] The advantages or beneficial effects of the technical solution of the present invention are:
[0043] The present invention constructs a fine-tuning training set based on historical quality assessment data of electronic data forensic appraisal opinions, and uses a low-order adaptive algorithm to supervise and fine-tune a large language model, thereby reducing the training cost while improving the model's pertinence and professionalism in the vertical field of quality assessment of electronic data forensic appraisal opinions. At the same time, based on prompt engineering, the efficiency and accuracy of quality assessment of forensic appraisal opinions are improved. In addition, based on retrieval enhancement generation technology, the knowledge in the latest appraisal standards is injected into the large language model to guide the quality assessment of appraisal opinions, which improves the timeliness and flexibility of the evaluation method and reduces the knowledge update cost of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 A flowchart of a method for evaluating the quality of electronic data forensic expert opinion in a preferred embodiment of the present invention;
[0045] Figure 2 This is a flow chart of step S1 in a preferred embodiment of the present invention;
[0046] Figure 3 This is a flow chart of step S2 in a preferred embodiment of the present invention;
[0047] Figure 4 This is a flow chart of step S3 in a preferred embodiment of the present invention;
[0048] Figure 5 This is a flow chart of step S4 in a preferred embodiment of the present invention;
[0049] Figure 6 This is a flow chart of step S42 in a preferred embodiment of the present invention;
[0050] Figure 7 This is a structural block diagram of a quality assessment system for electronic data forensic expert opinion in a preferred embodiment of the present invention;
[0051] Figure 8 It is a schematic diagram of the flow chart executed by the quality assessment module in a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0054] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.
[0055] The present invention aims to overcome the shortcomings of existing automatic text quality assessment technology and fill the gap in automatic text quality assessment methods for the field of forensic identification. A method and system for assessing the quality of electronic data forensic identification opinions based on Meta (formerly Facebook) artificial intelligence large language model (Large Language Model MetaAI, Llama) is designed and implemented.
[0056] In a preferred embodiment of the present invention, based on the above problems existing in the prior art, a method for evaluating the quality of electronic data forensic expert opinions based on the Llama model is provided. Figure 1 As shown, including:
[0057] Step S1, constructing training data based on the quality assessment data of the electronic data forensic appraisal opinion;
[0058] Step S2, according to the training data, a low-order adaptive algorithm is used to perform supervised fine-tuning training on the large language model to obtain a quality assessment model applied to the vertical field of quality assessment of judicial appraisal opinions; preferably, the large language model is a Llama model, such as a Llama3.1-70b-instruct model;
[0059] Step S3, judging whether the appraisal standards used in the judicial appraisal opinion to be evaluated are applicable and correct;
[0060] If yes, proceed to step S4;
[0061] If not, the judicial appraisal opinion to be evaluated is judged to be unqualified and the reasons are output;
[0062] Step S4, based on the retrieval enhancement generation model and the prompting project, the quality assessment model outputs the quality assessment result for the judicial appraisal opinion to be assessed.
[0063] Specifically, the present invention proposes a quality assessment method for electronic data forensic appraisal opinions based on Llama, which is specifically: filtering and desensitizing the quality assessment data of electronic data forensic appraisal opinions in previous years to obtain a fine-tuning training set, and using a low-order adaptive (LoRA) algorithm to supervise and fine-tune the open source Llama model, so that the Llama base model can basically understand the basic requirements of forensic appraisal texts and quality assessment; integrating expert knowledge, designing prompt words and corresponding judgment logic, and guiding the Llama model to meet the requirements of sequentiality, integrity and logic of electronic data forensic appraisal instruction evaluation; based on RAG technology and appraisal standard knowledge base, injecting the latest forensic expertise into large language models, thereby improving the timeliness and flexibility of the evaluation method.
[0064] The combination of the above model fine-tuning, RAG technology and prompt engineering improves the efficiency and accuracy of the Llama model in the vertical field of forensic appraisal opinion quality assessment, and effectively reduces the cost of model training and knowledge updating.
[0065] As a preferred embodiment, wherein Figure 2 As shown, step S1 includes:
[0066] Step S11, filtering the quality assessment data to obtain first screening data;
[0067] Step S12, desensitizing the first screening data to obtain second screening data, and using the second screening data as training data.
[0068] Specifically, based on the evaluation data accumulated from the electronic data forensic quality evaluation activities over the years, data filtering and data desensitization are performed, and the evaluation data after data filtering and data desensitization is used as training data. This training set can give the trained model knowledge in the specific field of electronic data forensic quality evaluation.
[0069] As a preferred implementation, in step S11, the quality assessment data is extracted based on preset filtering rules to obtain first screening data, which at least includes any one or more combinations of identification text content, expert evaluation scores, and expert evaluation opinions.
[0070] Specifically, the appraisal text content, expert evaluation score and expert evaluation opinion in the evaluation data are extracted based on the preset filtering rules. The expert evaluation opinion usually includes the judgment content of the order, completeness and logic of the appraisal text, and the corresponding evaluation score can be obtained based on the evaluation opinion.
[0071] As a preferred implementation, in step S12, sensitive information in the identification text is replaced with preset characters to obtain second screening data after data desensitization.
[0072] Specifically, the screened evaluation data is desensitized, specifically: the institution and personal identity information in the identification text is replaced with preset characters to obtain the desensitized evaluation data.
[0073] As a preferred implementation, the large language model is a Llama model.
[0074] As a preferred embodiment, wherein Figure 3 As shown, step S2 includes:
[0075] Step S21, segmenting and encoding the training data to obtain an identification opinion vector;
[0076] Step S22, freezing the first weight parameter of the large language model;
[0077] Step S23, adding a trainable second weight parameter to the cross attention layer of the large language model;
[0078] Step S24, training or updating the second weight parameter according to the identification opinion vector to obtain the trained or updated second weight parameter;
[0079] Step S25, merging the trained or updated second weight parameter with the frozen first weight parameter to obtain a merged weight parameter, and constructing a quality assessment model based on the merged weight parameter.
[0080] Specifically, in this embodiment, the Tiktoken open source word segmenter is used to segment and encode the training data. Then, based on the original Llama3.1-70b-instruct model, the low-order adaptive (LoRA) algorithm is used to fine-tune the model. Specifically, the original weights of the model are first frozen, that is, the reverse gradient update of the parameters of the specified list is set to FALSE. Next, trainable LoRA weights are added to the CrossAttention layer of the Llama3.1-70b-instruct model, and only the LoRA weights are trained and updated.
[0081] The trained LoRA weight parameters are merged with the frozen parameters of the original model to obtain the final large model applied to the vertical field of judicial appraisal opinion quality assessment. As a result, the fine-tuned model can be used to reason about new inputs.
[0082] As a preferred embodiment, wherein Figure 4 As shown, step S3 includes:
[0083] Step S31, extracting the entrusted matters and the appraisal standards used in the judicial appraisal opinion to be evaluated based on the quality assessment model;
[0084] Step S32, based on the pre-constructed electronic data judicial appraisal standard applicable rule library, determine whether the appraisal standards used in the judicial appraisal opinion to be evaluated are correctly applied and whether they meet the requirements of the entrusted matter.
[0085] Specifically, in this embodiment, a rule base for the application of electronic data judicial appraisal standards is constructed, and a fine-tuned Llama model is used to extract the entrusted matters and the appraisal standards used. Based on the rule base, it is judged whether the appraisal standards used in the appraisal opinion are correctly applicable and whether they can meet the requirements of the appraisal entrusted matters. If not, the appraisal opinion is directly judged as unqualified and the reasons for the deduction are output.
[0086] As a preferred embodiment, wherein Figure 5 As shown, step S4 includes:
[0087] Step S41, based on the identification standard terms used in the judicial identification opinion to be evaluated, the search enhancement generation model is called to obtain the identification standard related knowledge content, and the quality assessment model is used to summarize the identification steps based on the identification standard related knowledge content and the first prompt word;
[0088] Step S42, input the judicial appraisal opinion to be evaluated and the summarized appraisal steps into the prompt word template, use the quality assessment model to make a judgment, and output the judgment results of completeness, sequence and logic;
[0089] Step S43, calculate the evaluation score according to the deduction reasons and corresponding deduction scores in the integrity, sequentiality and logic judgment results, and then input the deduction reasons into the quality evaluation model to generate a quality evaluation result.
[0090] As a preferred embodiment, wherein Figure 6 As shown, step S42 includes:
[0091] Step S421, using the quality assessment model, outputting the integrity judgment result according to the second prompt word, the task description of the second prompt word includes the full text of the forensic expert opinion to be evaluated, the summarized appraisal steps, and the instructions for judging whether the appraisal process complies with the inspection steps required by the appraisal standard, and the output format includes the step number, answer, reason and the corresponding original content of the appraisal opinion;
[0092] Step S422, comparing the positions of the original contents of the appraisal opinions corresponding to the serial numbers of two adjacent steps in the judicial appraisal opinions to be evaluated with the operation sequence of the appraisal standards, and outputting the sequential judgment result;
[0093] Step S423, using the quality assessment model, outputs the logical judgment result based on the third prompt word. The task description of the third prompt word includes the full text of the judicial appraisal opinion to be evaluated and the quality of judging whether the appraisal process is consistent with the appraisal opinion and whether the appraisal opinion corresponds one-to-one with the entrusted matters. The output format includes the reason for deduction and the deduction score.
[0094] Specifically, firstly, based on the Retrieval Enhanced Generation (RAG) technology, the identification standard is searched according to the extracted identification standard name, and the corresponding identification standard full text and operation instructions are retrieved. Then, the prompt words are set, and the identification steps of the identification standard are summarized respectively using the fine-tuned Llama model, and the identification steps of the identification standard are stored in an array.
[0095] Exemplarily, the prompt words summarizing the identification steps are in the following form:
[0096] Task: Please summarize the inspection steps described in the text in brackets: {text}.
[0097] Output form: Step 1 - xxx; Step 2 - xxx; xxx; Step N - xxx.
[0098] Based on the prompt project, the integrity, sequence and logic of the identification steps are judged as follows:
[0099] Completeness judgment of identification steps: The full text of the identification opinion and the standard operating steps summarized in the fourth step are input into the prompt word template. The Llama model fine-tuned in the second step is used to judge whether the content of the identification opinion covers the standard requirements and whether the identification steps are completely operated, that is, the completeness judgment. The unfinished standard operating steps are recorded, and the original sentences of the identification opinion corresponding to the completed standard inspection steps are output. Among them, if the unfinished steps are necessary conditions, the identification opinion is directly judged as unqualified.
[0100] For example, the prompt word form for completeness judgment is as follows:
[0101] task: The content of the judicial appraisal opinion is: {text_A}.
[0102] The content of the inspection steps required by the judicial appraisal standards is: {text_B}.
[0103] Does the appraisal process of this appraisal opinion comply with the inspection steps required by the judicial appraisal standards?
[0104] Output form:
[0105] You need to answer and extract the corresponding original content in the following format, and output it in JSON format, which contains the following keys: step_id, answer, reason, content. step_id refers to the sequence number of the step, answer refers to the answer yes or no, reason refers to the reason, and content represents the original content of the corresponding appraisal opinion:
[0106] {"step_id":1,"answer":"Yes or No",reason":"Reason xxx","content":"The corresponding original content of the appraisal opinion xxx"},
[0107] {"step_id":2,"answer":"Yes or No",reason":"Reason xxx","content":"The corresponding original content of the appraisal opinion xxx"}'.
[0108] Determine the order of identification steps: Store the original sentences corresponding to the completed standard inspection steps as a string, that is, the content of "content". Use Python's find() function to determine the position of "content" in the entire identification opinion. n , that is, to locate the original text of the inspection steps. By comparing the positions of two adjacent inspection steps in the appraisal opinion, n With l n+1, determine whether the sequence of inspection steps follows the operation sequence of the identification standard. n > n+1 , indicating that the operation that should have been completed in step N is performed after step N+1, so step N does not conform to the operation sequence. The relevant information of all steps that do not conform to the operation sequence is stored, that is, the sequential judgment is completed.
[0109] Logical judgment of the appraisal steps: Conduct a logical judgment on the appraisal opinion. Input the appraisal opinion into the prompt word template and use the Llama model fine-tuned in the second step to judge the logic from subjective aspects such as whether the appraisal process of the appraisal opinion is consistent with the appraisal opinion and whether the appraisal opinion corresponds to the entrusted matters one by one, and obtain the deduction score and reasons.
[0110] For example, the logical judgment prompt words are in the following form:
[0111] Task: The content of the judicial appraisal opinion is: {text_A}.
[0112] Please judge the logic of the appraisal opinion from the aspects of whether the appraisal process is consistent with the appraisal opinion, whether the appraisal opinion corresponds to the entrusted matters, etc., and give the deduction points and reasons. The total deduction shall not exceed {custom score} points.
[0113] Output form:
[0114] You need to answer and explain the reason in the following format, and the output should be in JSON format, which contains the following keys: score, reason. Score refers to the deduction score, and reason refers to the reason: {"score":xx,reason":"reasonxxx"}
[0115] Finally, generate the evaluation score and evaluation opinion. After the above-mentioned completeness, sequentiality and logic judgment, count the unfinished standard operation steps, steps that do not conform to the operation sequence, deduction reasons and customized deduction values, combine the deduction values and reasons in the logical judgment, calculate the final evaluation score, and input the deduction reasons into the Llama model after fine-tuning in the second step, and use this vertical field large model to generate evaluation opinions.
[0116] For example, the scoring prompt word form is as follows:
[0117] Task: The full score is 100 points. Given an appraisal opinion: In the completeness judgment, the deduction score is {socre1}, and the reason is {reason1}. In the sequential judgment, the deduction score is {socre2}, and the reason is {reason2}. In the logical judgment, the deduction score is {socre3}, and the reason is {reason3}.
[0118] Output form:
[0119] The total score of the appraisal opinion needs to be calculated and the reasons need to be stated in the following format. It is required to be output in JSON format, which contains the following keys: score, reason. Score refers to the final total score, and reason refers to the reason for the deduction: {"score":xx,"reason":"reason xxx"}
[0120] The present invention also provides a quality assessment system for electronic data forensic expert opinion based on the Llama model, which is used to implement the quality assessment method for electronic data forensic expert opinion based on the Llama model as described above. Figure 7 As shown, the system is divided into three parts: training data construction module 1, model training module 2 and quality assessment module 3, among which:
[0121] The training data construction module 1 is used to construct training data according to the quality assessment data of the electronic data forensic appraisal opinion;
[0122] Model training module 2, connected to training data building module 1, is used to perform supervised fine-tuning training on a large language model using a low-order adaptive algorithm based on the training data, so as to obtain a quality assessment model applied to the vertical field of quality assessment of judicial appraisal opinions;
[0123] The quality assessment module 3 is connected to the model training module 2 to perform the following steps: Figure 8 The steps shown are to determine whether the appraisal standards used in the judicial appraisal opinions to be evaluated are correctly applied; if they are wrong, the judicial appraisal opinions to be evaluated are judged to be unqualified and the reasons are output; when the appraisal standards are correctly applied, based on the retrieval enhancement generation model and prompt engineering, the quality assessment model outputs the quality assessment results for the judicial appraisal opinions to be evaluated.
[0124] More implementation details of the system and the method have been disclosed and will not be repeated in this embodiment.
[0125] In view of the large number of electronic data forensic appraisal cases in the prior art, it is time-consuming and laborious to rely on manual methods to conduct quality assessment of the appraisal process and appraisal opinions. Although existing technologies can realize automated quality assessment of laboratory reports and machine-generated texts, these technologies cannot fully understand the forensic appraisal opinions and cannot meet the requirements of forensic appraisal quality assessment when applied to forensic appraisal quality assessment, and therefore cannot fully assess the true quality of forensic appraisal opinions. Therefore, the present invention aims to develop an automated method and system for the quality assessment of electronic data forensic appraisal opinions, improve the accuracy of quality assessment methods in the field of forensic appraisal, and improve the efficiency of forensic appraisal quality assessment.
[0126] In view of the fast update and iteration speed of existing judicial appraisal standards, methods and appraisal quality assessment standards, the long text of appraisal opinions and the many professional terms, the existing technology usually requires the manual establishment of large-scale data sets to train deep learning models, and the computing power and time cost of training are large. The method and system of the present invention use the method of combining model fine-tuning, retrieval enhancement generation (RAG) technology and prompt engineering to establish a fine-tuning data set using the electronic data judicial appraisal assessment data accumulated over the years, only train some parameters of the Llama model, and design prompt words and corresponding judgment logic to meet the sequentiality, integrity and logic requirements of the quality assessment of judicial appraisal opinions, which greatly reduces the cost of model construction. Retrieval enhancement generation technology can inject the latest appraisal knowledge into the Llama model, without the need to retrain the model when the knowledge is updated, reducing the cost of model training and knowledge updating.
[0127] The existing large language model can understand the semantics of general text, but lacks forensic expertise, and forensic expertise is updated and iterated quickly. The method and system of the present invention are based on large language model technology and retrieval enhancement generation technology, and a special knowledge base and evaluation rules are designed to improve the professionalism and timeliness of the evaluation method.
[0128] The advantages or beneficial effects of adopting the above technical solution are:
[0129] (1) The present invention constructs a fine-tuning training set based on the quality assessment data of judicial appraisals over the years, and uses the low-order adaptive (LoRA) algorithm to supervise and fine-tune the open source large-scale language model Llama3.1-70b-instruct, thereby reducing the training cost and improving the pertinence and professionalism of the Llama model in the vertical field of quality assessment of electronic data judicial appraisal opinions;
[0130] (2) By integrating expert knowledge with the needs of judicial appraisal quality assessment, we designed prompt words and corresponding algorithm logic to judge completeness, sequence, and logic, thereby improving the efficiency and accuracy of judicial appraisal opinion quality assessment;
[0131] (3) Based on the retrieval-augmented generation (RAG) technology, the knowledge of the latest appraisal standards is injected into the large model to guide the quality assessment of appraisal opinions, which improves the timeliness and flexibility of the evaluation method and reduces the knowledge updating cost of the model.
[0132] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of this specification and illustrations should be included in the protection scope of the present invention.
Claims
1. A method for evaluating the quality of electronic data forensic expert opinion based on the Llama model, characterized in that: include: Step S1, constructing training data based on the quality assessment data of the electronic data forensic appraisal opinion; Step S2, based on the training data, using a low-order adaptive algorithm to perform supervised fine-tuning training on the large language model to obtain a quality assessment model applied to the vertical field of quality assessment of judicial appraisal opinions; Step S3, judging whether the appraisal standards used in the judicial appraisal opinion to be evaluated are applicable and correct; If yes, proceed to step S4; If not, the judicial appraisal opinion to be evaluated is judged to be unqualified and the reasons are output; Step S4, based on the retrieval enhancement generation model and the prompting project, the quality assessment model outputs a quality assessment result for the judicial appraisal opinion to be assessed.
2. The method according to claim 1, characterized in that: The step S1 comprises: Step S11, filtering the quality assessment data to obtain first screening data; Step S12: desensitize the first screening data to obtain second screening data, and use the second screening data as the training data.
3. The method according to claim 2, characterized in that In the step S11, the quality assessment data is extracted based on preset filtering rules to obtain the first screening data, which at least includes any one or more combinations of identification text content, expert evaluation scores, and expert evaluation opinions.
4. The method according to claim 2, characterized in that In the step S12, the sensitive information in the identification text is replaced with preset characters to obtain the second screening data after data desensitization.
5. The method according to claim 1, characterized in that: The large language model is a Llama model.
6. The method according to claim 1, characterized in that The step S2 comprises: Step S21, segmenting and encoding the training data to obtain an identification opinion vector; Step S22, freezing the first weight parameter of the large language model; Step S23, adding a trainable second weight parameter to the cross attention layer of the large language model; Step S24, training or updating the second weight parameter according to the identification opinion vector to obtain the trained or updated second weight parameter; Step S25, merging the trained or updated second weight parameter with the frozen first weight parameter to obtain a merged weight parameter, and constructing the quality assessment model based on the merged weight parameter.
7. The method according to claim 1, characterized in that The step S3 comprises: Step S31, extracting the entrusted matters and the appraisal standards used in the judicial appraisal opinion to be evaluated based on the quality assessment model; Step S32, based on the pre-constructed electronic data judicial appraisal standard applicable rule library, determine whether the appraisal standard used in the judicial appraisal opinion to be evaluated is correctly applied and whether it meets the requirements of the entrusted matter.
8. The method according to claim 1, characterized in that The step S4 comprises: Step S41, based on the identification standard terms used in the judicial identification opinion to be evaluated, calling the retrieval enhancement generation model to obtain the identification standard related knowledge content, and using the quality assessment model to summarize the identification steps based on the identification standard related knowledge content and the first prompt word; Step S42, inputting the judicial appraisal opinion to be evaluated and the appraisal steps summarized into the prompt word template, using the quality evaluation model to make a judgment, and outputting the judgment results of completeness, sequence and logic; Step S43, calculating the evaluation score according to the deduction reasons and the corresponding deduction scores in the integrity, sequentiality and logic judgment results, and then inputting the deduction reasons into the quality assessment model to generate the quality assessment result.
9. The method according to claim 1, characterized in that: The step S42 comprises: Step S421, using the quality assessment model, outputting the integrity judgment result according to the second prompt word, wherein the task description of the second prompt word includes the full text of the forensic expert opinion to be evaluated, the summarized appraisal steps, and instructions for judging whether the appraisal process complies with the inspection steps required by the appraisal standard, and the output format includes the step number, answer, reason and the corresponding original content of the appraisal opinion; Step S422, comparing the positions of the original contents of the appraisal opinions corresponding to the serial numbers of two adjacent steps in the judicial appraisal opinion to be evaluated with the operation sequence of the appraisal standards, and outputting a sequential judgment result; Step S423, using the quality assessment model, output a logical judgment result based on the third prompt word. The task description of the third prompt word includes the full text of the judicial appraisal opinion to be evaluated and the quality of judging whether the appraisal process is consistent with the appraisal opinion and whether the appraisal opinion corresponds one-to-one with the entrusted matters. The output format includes the reason for deduction and the deduction score.
10. A quality assessment system for electronic data forensic expert opinion based on the Llama model, characterized in that: The method for implementing the quality assessment method of electronic data forensic expert opinion based on the Llama model as described in any one of claims 1 to 9 comprises: A training data construction module, used to construct training data based on quality assessment data of electronic data forensic appraisal opinions; A model training module, connected to the training data building module, is used to perform supervised fine-tuning training on a large language model using a low-order adaptive algorithm based on the training data to obtain a quality assessment model applied to the vertical field of quality assessment of judicial appraisal opinions; The quality assessment module is connected to the model training module and is used to determine whether the appraisal standards used in the judicial appraisal opinions to be evaluated are correctly applied; if they are wrong, the judicial appraisal opinions to be evaluated are judged to be unqualified and the reasons are output; when the appraisal standards are correctly applied, the quality assessment model outputs the quality assessment results for the judicial appraisal opinions to be evaluated based on the retrieval enhancement generation model and prompt engineering.