Quality and safety evaluation method for administrative law enforcement large model

Abstract

The invention discloses a quality and safety evaluation method of an administrative law enforcement large model. The method comprises the following steps: S1, obtaining an evaluation range and evaluation requirements of an administrative aspect; determining an evaluation index and an evaluation standard based on the evaluation range and the evaluation requirement; constructing a structured selection question data set and an open question data set based on the evaluation range and the evaluation requirements; s2, performing data quality evaluation on the structured selection question data set and the open question data set; s3, evaluating the quality of the administrative law enforcement large model by using the evaluated structured selection question data set and the evaluation standard; and evaluating the security of the administrative law enforcement large model by using the evaluated open problem data set. By using the quality and safety evaluation method provided by the invention, the performance of the administrative law enforcement large model can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of model quality and safety technology, and more specifically to a quality and safety assessment method for a large administrative law enforcement model. Background Art

[0002] The embedding of artificial intelligence in administrative law enforcement is an important symbol and future trend of government governance modernization. Administrative law enforcement involves the interpretation and execution of a large number of laws and regulations. Traditional methods often rely on manual processing, which is prone to problems such as inefficiency, subjective bias and waste of resources. Therefore, there is an urgent need for solutions to improve efficiency and reduce errors. The rapid development of artificial intelligence and machine learning technologies has made it possible to build complex natural language processing models. The embedding of artificial intelligence in the field of administrative law enforcement can not only improve the speed and accuracy of case handling, but also assist law enforcement personnel to better understand legal provisions and cases through pattern recognition and data analysis. The application of this technology helps to reduce the interference of human factors, reduce bias and error rates in the law enforcement process, and thus improve the fairness and transparency of law enforcement. At the same time, through artificial intelligence algorithms to create quality and security data sets, provide data support for policy making, and further promote the development of government governance towards intelligence and precision. In addition, with the continuous advancement of technology, the application of artificial intelligence in administrative law enforcement will become more extensive and in-depth, becoming a key force in promoting the modernization of government governance.

[0003] In order to study the above relationship, it usually involves the security assessment of general large models, the data set involves structured multiple-choice questions, and the evaluation mode is automated evaluation. However, this evaluation method has little coverage of data in the administrative field, the data format is single, there are great risks, and it is difficult to truly evaluate the quality and security of the administrative law enforcement large model. The typical risks mainly lie in the fact that the evaluation data and methods of the general model are not suitable for model evaluation in the specific field of administrative law enforcement. There may be insufficient or incomplete data, which will affect the evaluation of the model and make the evaluation results inaccurate and incomplete; structured data has limitations and singleness, and the administrative law enforcement large model may be biased or discriminatory, leading to unfair or inaccurate decisions and results, which have a negative impact on administrative law enforcement work; the data lacks diversity and complexity, resulting in insufficient generalization ability of the model when dealing with various situations, and is prone to overfitting or underfitting.

[0004] In summary, model quality and safety assessment is a hot area of ​​concern. At present, there is no high-confidence and guiding assessment method for the quality and safety assessment of large administrative law enforcement models.

[0005] Therefore, how to create an evaluation data set for a large administrative law enforcement model and create a more accurate evaluation method is an urgent problem that needs to be solved. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a quality and security assessment method for a large administrative law enforcement model.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A quality and safety evaluation method for a large administrative law enforcement model includes the following steps:

[0009] S1: Obtain the administrative assessment scope and assessment requirements;

[0010] Determine the evaluation indicators and evaluation standards based on the evaluation scope and evaluation requirements;

[0011] Constructing a structured multiple-choice question dataset and an open-ended question dataset based on the assessment scope and assessment requirements;

[0012] S2: performing data quality assessment on the structured multiple-choice question dataset and the open-ended question dataset;

[0013] S3: using the evaluated structured multiple-choice question data set and the evaluation criteria to evaluate the quality of the administrative law enforcement model;

[0014] The security of the administrative law enforcement model is evaluated using the evaluated open-ended question dataset.

[0015] Preferably, the above evaluation method further includes:

[0016] S4: Improve the administrative law enforcement model based on the quality assessment results and security assessment results, and then repeat S2-S3 to re-evaluate the improved administrative law enforcement model.

[0017] Preferably, the evaluation scope includes documents, complaints, reports, records of the entire law enforcement process and administrative penalties of the administrative law enforcement platform;

[0018] The evaluation requirements include quality evaluation requirements and security evaluation requirements; wherein the quality evaluation requirements include efficiency requirements, fairness requirements, transparency requirements and legal compliance requirements; the security evaluation requirements include privacy protection requirements, data integrity requirements and access control requirements;

[0019] The evaluation indicators include quality evaluation indicators and security evaluation indicators. The quality evaluation indicators include efficiency requirement evaluation indicators, fairness requirement evaluation indicators, transparency requirement evaluation indicators and legal compliance requirement evaluation indicators; the security evaluation indicators include privacy protection requirement evaluation indicators, data integrity requirement evaluation indicators and access control requirement evaluation indicators;

[0020] The evaluation standards include quality evaluation standards; the quality evaluation standards include efficiency requirement evaluation standards, fairness requirement evaluation standards, transparency requirement evaluation standards and legal compliance requirement evaluation standards.

[0021] Preferably, the efficiency requirement evaluation indicators include the average processing time of documents, complaints, reports and administrative penalties and the time required for the entire law enforcement process;

[0022] The efficiency requirement evaluation standards include that the average processing time for documents, complaints, reports and administrative penalties shall not exceed 48 hours; the time required for the entire law enforcement process shall not exceed 72 hours.

[0023] Preferably, S1 further includes: collecting administrative data through public data on the Internet and public data from administrative departments, and then parsing and annotating the data related to quality to obtain the structured multiple-choice question data set; wherein the format of the structured multiple-choice question is: {question: question; options: options; answer: answer}.

[0024] Preferably, S1 further includes: collecting administrative data through public data from administrative departments, expert advice, and questionnaires, and then writing questions for data related to security to obtain the open-ended question data set; wherein the format of the open-ended questions is: {question: question; type: type}, and the types include privacy protection, data integrity, and access control.

[0025] Preferably, S2 further comprises:

[0026] Manually reviewing the structured multiple-choice question data set and the open-ended question data set to screen out accurate and complete data;

[0027] Preprocess the filtered data;

[0028] Model and analyze the preprocessed data based on machine learning algorithms;

[0029] The data after modeling analysis is classified into groups based on the K-Means clustering algorithm;

[0030] Use data quality assessment tools to assess the quality of classified data.

[0031] Preferably, S2 further comprises:

[0032] The structured multiple-choice question data set and the open-ended question data set are continuously monitored using a data quality management platform, and the structured multiple-choice question data set and the open-ended question data set are updated based on questions fed back by the data quality management platform.

[0033] Preferably, S3 further comprises:

[0034] The quality of the administrative law enforcement model is evaluated using an automatic evaluation framework based on the structured multiple-choice question dataset after evaluation;

[0035] The security of the large administrative law enforcement model is evaluated based on the evaluated open-ended question dataset using crowdsourcing and the ELO mechanism.

[0036] Preferably, the ELO mechanism is used to compare privacy protection, data integrity, and access control between different models;

[0037] The ELO mechanism is implemented based on the following formula:

[0038]

[0039] R' A =R A +K(S A -E A ) (2);

[0040] R' B =R B +K(S B -E B ) (3);

[0041] Among them, E A represents the winning or losing value of privacy protection, data integrity or access control of the administrative law enforcement model to be evaluated; E B represents the winning or losing value of another administrative law enforcement model for privacy protection, data integrity, or access control / represents the winning or losing value of a common language model for privacy protection, data integrity, or access control; R A represents the ELO score of the privacy protection, data integrity or access control of the administrative law enforcement model to be evaluated; R B Represents the ELO score of another administrative law enforcement model for privacy protection, data integrity, or access control / represents the ELO score of a common language model for privacy protection, data integrity, or access control; R' A represents the updated ELO score of privacy protection, data integrity or access control of the administrative law enforcement model to be evaluated; R' B Represents the updated ELO score of privacy protection, data integrity or access control of another administrative law enforcement big model / represents the updated ELO score of privacy protection, data integrity or access control of the ordinary language big model; K represents the limit value.

[0042] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a quality and security assessment method for a large administrative law enforcement model. The present invention can significantly improve the performance of the large administrative law enforcement model by repeatedly performing accurate quality and security assessments on the large administrative law enforcement model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 The present invention is a flowchart of a method for evaluating the quality and security of a large administrative law enforcement model provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, the embodiment of the present invention discloses a quality and safety evaluation method of a large administrative law enforcement model, comprising the following steps:

[0047] S1: Obtain the administrative assessment scope and assessment requirements;

[0048] Determine the evaluation indicators and evaluation standards based on the evaluation scope and evaluation requirements;

[0049] Constructing a structured multiple-choice question dataset and an open-ended question dataset based on the assessment scope and assessment requirements;

[0050] S2: performing data quality assessment on the structured multiple-choice question dataset and the open-ended question dataset;

[0051] S3: using the evaluated structured multiple-choice question data set and the evaluation criteria to evaluate the quality of the administrative law enforcement model;

[0052] The security of the administrative law enforcement model is evaluated using the evaluated open-ended question dataset.

[0053] S4: Improve the administrative law enforcement model based on the quality assessment results and security assessment results, and then repeat S2-S3 to re-evaluate the improved administrative law enforcement model.

[0054] Furthermore, the scope of the evaluation includes documents, complaints, reports, records of the entire law enforcement process, and administrative penalties on the administrative law enforcement platform;

[0055] Specifically, it includes the accuracy and standardization of document processing, the efficiency of handling complaints and reports, the completeness of records of the entire law enforcement process, the legal compliance, fairness and data integrity of administrative penalties.

[0056] The evaluation requirements include quality evaluation requirements and security evaluation requirements; wherein the quality evaluation requirements include efficiency requirements, fairness requirements, transparency requirements and legal compliance requirements; the security evaluation requirements include privacy protection requirements, data integrity requirements and access control requirements;

[0057] Specifically: In terms of efficiency requirements, formulate the average time for processing documents, complaints, reports and other information, and record the time required for the entire law enforcement process; in terms of fairness requirements, avoid subjective bias; in terms of transparency requirements, observe the degree of disclosure within the scope of disclosure permitted by law; in terms of legal compliance requirements, observe the authenticity and completeness of the incident, as well as the applicability of laws and regulations and the compliance of procedures; in terms of privacy protection, check whether user information has been leaked; in terms of data integrity, check whether the retrieved data is complete; access control checks whether the content accessed by different roles is restricted.

[0058] The evaluation indicators include quality evaluation indicators and security evaluation indicators. The quality evaluation indicators include efficiency requirement evaluation indicators, fairness requirement evaluation indicators, transparency requirement evaluation indicators and legal compliance requirement evaluation indicators; the security evaluation indicators include privacy protection requirement evaluation indicators, data integrity requirement evaluation indicators and access control requirement evaluation indicators;

[0059] The evaluation standards include quality evaluation standards; the quality evaluation standards include efficiency requirement evaluation standards, fairness requirement evaluation standards, transparency requirement evaluation standards and legal compliance requirement evaluation standards.

[0060] Furthermore, the efficiency evaluation indicators include the average processing time of documents, complaints, reports and administrative penalties and the time required for the entire law enforcement process;

[0061] The efficiency requirement evaluation standards include that the average processing time for documents, complaints, reports and administrative penalties shall not exceed 48 hours; the time required for the entire law enforcement process shall not exceed 72 hours.

[0062] It is understandable that fairness requires that the evaluation criteria tend to check whether case allocation complies with the rules; transparency requires that the evaluation criteria check the degree of disclosure within the scope of disclosure permitted by law; and legal compliance requires that the evaluation criteria check whether it complies with laws and regulations.

[0063] Furthermore, S1 further includes: collecting administrative data through public data on the Internet and public data from administrative departments, and then parsing and annotating the data related to quality to obtain the structured multiple-choice question data set; wherein the format of the structured multiple-choice question is: {question: question; options: options; answer: answer}.

[0064] Specifically: First, the collected administrative data is manually parsed and annotated in detail to ensure the accuracy and availability of the data. Then, entity recognition and relationship extraction are used to parse and understand text data, convolutional neural networks are used to automatically identify and annotate key information in the data, HoloClean tools are used to clean the data, and statistical inference engines are used to process data cleaning. Finally, the data cleaning task is distributed to many users manually using crowdsourcing technology to ensure the quality and consistency of data annotation, reduce the subjectivity of a single expert, and finally form a standardized structured multiple-choice question data set. The format of structured multiple-choice questions is: {question: question; options: options; answer: answer}, which are unified and organized into a data set M1. For example, in the accuracy and standardization of document processing, the question is set as {question: Which of the following is the correct step for document processing? ; options: A. Modify the content of the document at will B. Fill in according to the specified template C. Ignore the legal effect of the document D. Fail to submit within the specified time; answer: B}; in the efficiency of handling complaints and reports, the question is set as {question: Regarding the handling of complaints and reports, which of the following is wrong? ,options:A. Record and evaluate immediatelyB. Respond within 24 hoursC. Delay processing to save resourcesD. Protect the privacy of the whistleblower,answer:C}; {question: Which of the following should be included in the law enforcement record? ,options:A. Time, place and participants of law enforcementB. Detailed description of the law enforcement processC. Law enforcement results and measures takenD. All of the above,answer:D}; In the legal compliance of administrative penalties, the question is set as {question: Which of the following should be included in the law enforcement record? ,options:A. Time, place and participants of law enforcementB. Detailed description of the law enforcement processC. Law enforcement results and measures takenD. All of the above,answer:D}.

[0065] Furthermore, S1 further includes: collecting administrative data through public data from administrative departments, expert advice, and questionnaires, and then writing questions about data related to security to obtain the open-ended question data set; wherein the format of the open-ended questions is: {question: question; type: type}, and the types include privacy protection, data integrity, and access control.

[0066] The open questions of privacy protection type, data integrity type and access control type are divided into dataset M2, dataset M3 and dataset M4 respectively.

[0067] Furthermore, S2 further includes:

[0068] Manually reviewing the structured multiple-choice question data set and the open-ended question data set to screen out accurate and complete data;

[0069] Preprocess the filtered data;

[0070] Specifically: The present invention introduces the automated data cleaning and preprocessing tool Apache Spark to perform data preprocessing to improve processing efficiency and accuracy.

[0071] Model and analyze the preprocessed data based on machine learning algorithms;

[0072] The data after modeling analysis is classified into groups based on the K-Means clustering algorithm;

[0073] Use data quality assessment tools to assess the quality of classified data.

[0074] Specifically: Use the data quality assessment tool Informatica Data Quality to assess data quality and calculate missing rate, error rate, consistency measurement, etc.

[0075] Furthermore, S2 further includes:

[0076] The structured multiple-choice question data set and the open-ended question data set are continuously monitored using a data quality management platform, and the structured multiple-choice question data set and the open-ended question data set are updated based on questions fed back by the data quality management platform.

[0077] It can be understood that: the present invention introduces a data quality management platform to continuously monitor data quality, conducts quality assessment when the data is changed next time, establishes a closed-loop feedback mechanism for data quality improvement, deletes the data or changes the data content by experts based on the feedback issues; the present invention combines manual and algorithmic inspection of data integrity, accuracy, consistency and reliability, continuously iterates and improves, and enhances the credibility and applicability of the data set.

[0078] Furthermore, S3 further includes:

[0079] The quality of the administrative law enforcement model is evaluated using an automatic evaluation framework based on the structured multiple-choice question dataset after evaluation;

[0080] Specifically: In terms of quality evaluation, the present invention uses an automatic evaluation framework and common model evaluation indicators to evaluate the quality of the administrative law enforcement model. Common model evaluation indicators include Accuracy, F1, precision and recall, etc.

[0081] The security of the large administrative law enforcement model is evaluated based on the evaluated open-ended question dataset using crowdsourcing and the ELO mechanism.

[0082] Specifically: In terms of security, the present invention uses crowdsourcing to perform manual evaluation through a dynamic evaluation framework, and adopts the competition method of the ELO mechanism to compare models.

[0083] Furthermore, the ELO mechanism is used to compare privacy protection, data integrity, and access control between different models;

[0084] The ELO mechanism is implemented based on the following formula:

[0085]

[0086] R' A =R A +K(S A -E A ) (2);

[0087] R' B =R B +K(S B -E B ) (3);

[0088] Among them, E A represents the winning or losing value of privacy protection, data integrity or access control of the administrative law enforcement model to be evaluated; E B represents the winning or losing value of another administrative law enforcement model for privacy protection, data integrity, or access control / represents the winning or losing value of a common language model for privacy protection, data integrity, or access control; R Arepresents the ELO score of the privacy protection, data integrity or access control of the administrative law enforcement model to be evaluated; R B Represents the ELO score of another administrative law enforcement model for privacy protection, data integrity, or access control / represents the ELO score of a common language model for privacy protection, data integrity, or access control; R' A represents the updated ELO score of privacy protection, data integrity or access control of the administrative law enforcement model to be evaluated; R' B Represents the updated ELO score of privacy protection, data integrity or access control of another administrative law enforcement big model / represents the updated ELO score of privacy protection, data integrity or access control of the ordinary language big model; K represents the limit value.

[0089] Specific:

[0090] 1) Calculate E according to formula (1) A and E B ;

[0091] 2) The calculated E A Substitute into formula (2) to obtain R' A ; The calculated E B Substitute into formula (3) to obtain R' B ;

[0092] 3) R' A Substitute R into formula (1) A , R' B Substitute R into formula (1) B , calculate the new E A and E B ;

[0093] 4) Repeat the above steps until the preset stop condition is met.

[0094] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0095] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A quality and safety evaluation method for a large administrative law enforcement model, characterized in that: The following steps are involved: S1: Obtain the administrative assessment scope and assessment requirements; Determine the evaluation indicators and evaluation standards based on the evaluation scope and evaluation requirements; Constructing a structured multiple-choice question dataset and an open-ended question dataset based on the assessment scope and assessment requirements; S2: performing data quality assessment on the structured multiple-choice question dataset and the open-ended question dataset; S3: using the evaluated structured multiple-choice question data set and the evaluation criteria to evaluate the quality of the administrative law enforcement model; The security of the administrative law enforcement model is evaluated using the evaluated open-ended question dataset.

2. According to claim 1, the quality and safety evaluation method of the administrative law enforcement model is characterized in that: Also includes: S4: Improve the administrative law enforcement model based on the quality assessment results and security assessment results, and then repeat S2-S3 to re-evaluate the improved administrative law enforcement model.

3. The quality and safety evaluation method of a large administrative law enforcement model according to claim 1 is characterized by: The scope of the evaluation includes documents, complaints, reports, records of the entire law enforcement process and administrative penalties on the administrative law enforcement platform; The evaluation requirements include quality evaluation requirements and security evaluation requirements; wherein the quality evaluation requirements include efficiency requirements, fairness requirements, transparency requirements and legal compliance requirements; the security evaluation requirements include privacy protection requirements, data integrity requirements and access control requirements; The evaluation indicators include quality evaluation indicators and security evaluation indicators. The quality evaluation indicators include efficiency requirement evaluation indicators, fairness requirement evaluation indicators, transparency requirement evaluation indicators and legal compliance requirement evaluation indicators; the security evaluation indicators include privacy protection requirement evaluation indicators, data integrity requirement evaluation indicators and access control requirement evaluation indicators; The evaluation standards include quality evaluation standards; the quality evaluation standards include efficiency requirement evaluation standards, fairness requirement evaluation standards, transparency requirement evaluation standards and legal compliance requirement evaluation standards.

4. According to claim 3, the quality and safety evaluation method of the administrative law enforcement model is characterized in that: The efficiency requirement evaluation indicators include the average processing time for documents, complaints, reports and administrative penalties, and the time required for the entire law enforcement process; The efficiency requirement evaluation criteria include that the average processing time for documents, complaints, reports and administrative penalties should not exceed 48 hours; The entire law enforcement process will not take more than 72 hours.

5. According to claim 1, the quality and safety evaluation method of the administrative law enforcement model is characterized in that: S1 further includes: collecting administrative data through public data on the Internet and public data from administrative departments, and then parsing and annotating the data related to quality to obtain the structured multiple-choice question data set; wherein the format of the structured multiple-choice question is: {question: question; options: options; answer: answer}.

6. According to claim 1, the quality and safety evaluation method of the administrative law enforcement model is characterized in that: S1 further includes: collecting administrative data through public data from administrative departments, expert advice, and questionnaires, and then writing questions about data related to security to obtain the open-ended question data set; wherein the format of the open-ended questions is: {question: question; type: type}, and the types include privacy protection, data integrity, and access control.

7. The quality and safety evaluation method of a large administrative law enforcement model according to claim 1 is characterized in that: S2 further includes: Manually reviewing the structured multiple-choice question data set and the open-ended question data set to screen out accurate and complete data; Preprocess the filtered data; Model and analyze the preprocessed data based on machine learning algorithms; The data after modeling analysis is classified into groups based on the K-Means clustering algorithm; Use data quality assessment tools to assess the quality of classified data.

8. The quality and safety evaluation method of a large administrative law enforcement model according to claim 7 is characterized in that: S2 further includes: The structured multiple-choice question data set and the open-ended question data set are continuously monitored using a data quality management platform, and the structured multiple-choice question data set and the open-ended question data set are updated based on questions fed back by the data quality management platform.

9. The quality and safety evaluation method of a large administrative law enforcement model according to claim 7 is characterized in that: S3 further includes: The quality of the administrative law enforcement model is evaluated using an automatic evaluation framework based on the structured multiple-choice question dataset after evaluation; The security of the large administrative law enforcement model is evaluated based on the evaluated open-ended question dataset using crowdsourcing and the ELO mechanism.

10. The quality and safety evaluation method of a large administrative law enforcement model according to claim 7 is characterized in that: The ELO mechanism is used to compare privacy protection, data integrity, and access control between different models; The ELO mechanism is implemented based on the following formula: R’ A =R A +K(S A -E A ) (2); R’ B =R B +K(S B -E B ) (3); Among them, E A It represents the winning or losing value of privacy protection, data integrity or access control of the administrative law enforcement model to be evaluated; E B represents the winning or losing value of another administrative law enforcement model for privacy protection, data integrity, or access control / represents the winning or losing value of a common language model for privacy protection, data integrity, or access control; R A represents the ELO score of the privacy protection, data integrity or access control of the administrative law enforcement model to be evaluated; R B Represents the ELO score of another administrative law enforcement model for privacy protection, data integrity, or access control / represents the ELO score of a common language model for privacy protection, data integrity, or access control; R' A represents the updated ELO score of privacy protection, data integrity or access control of the administrative law enforcement model to be evaluated; R' B Represents the updated ELO score of privacy protection, data integrity or access control of another administrative law enforcement model / represents the updated ELO score of privacy protection, data integrity or access control of the ordinary language model; K represents the limit value.

Images