Data compliance detection method and system based on large language model and ai-agent

Abstract

The application discloses a data compliance detection method and system based on a large language model and an AI-Agent, relates to the technical field of data detection, and comprises the following steps: collecting data from multiple data sources automatically through an AI-Agent, performing preprocessing and standardization on the collected data, performing historical performance analysis on the standardized data, and optimizing the data collection strategy and data preprocessing strategy of the AI-Agent based on the analysis result; performing feature extraction and feature optimization on the preprocessed data through an LLaMA-2 model; the LLaMA-2 model performs deep learning analysis on the optimized features, and an artificial person fine-tunes the parameters of the LLaMA-2 model based on the analysis result; the AI-Agent automatically acquires the latest compliance rules through a rule engine, then calls the LLaMA-2 model, performs compliance evaluation on the preprocessed data based on the acquired compliance rules, and further generates a compliance report. The application can realize the automation and high efficiency of data compliance detection, reduce the demand for artificial intervention, and improve the accuracy and adaptability of data compliance detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data detection, and in particular to a data compliance detection method and system based on a large language model and an AI-Agent. BACKGROUND

[0002] The prior art mainly relies on manual processes or basic automated tools for data monitoring, which not only inefficient, but also difficult to adapt to changing compliance requirements. In addition, existing solutions often cannot efficiently process and analyze large-scale, complex data sets, resulting in inaccurate compliance risk assessments. SUMMARY

[0003] The present application provides a data compliance detection method and system based on a large language model and an AI-Agent to improve the accuracy and speed of data compliance detection, while reducing the need for human intervention.

[0004] In a first aspect, the present application provides a data compliance detection method based on a large language model and an AI-Agent, which solves the above technical problems by adopting the following technical solutions:

[0005] A data compliance detection method based on a large language model and an AI-Agent, comprising the following steps:

[0006] S1, through the AI-Agent, automatically collecting data from multiple data sources, performing preprocessing and standardization on the collected data, performing historical performance analysis on the standardized data, and optimizing the data collection strategy and data preprocessing strategy of the AI-Agent based on the analysis results;

[0007] S2, through the LLaMA-2 model, performing feature extraction and feature optimization on the preprocessed data;

[0008] S3, the LLaMA-2 model performs deep learning analysis on the optimized features, and the human adjusts the parameters of the LLaMA-2 model based on the analysis results;

[0009] S4, the AI-Agent automatically obtains the latest compliance rules through the rule engine, then calls the LLaMA-2 model, and performs compliance evaluation on the preprocessed data based on the obtained compliance rules, and further generates a compliance report.

[0010] Optionally, in step S1, the AI-Agent performs preprocessing, standardization, and historical performance analysis on the collected data, specifically including:

[0011] According to the pre-set cleaning rules, the AI-Agent performs data cleaning preprocessing operations on the collected data;

[0012] According to the pre-set standardized rules, the AI-Agent performs format standardization on the data after cleaning;

[0013] The AI-Agent performs historical performance analysis on the data after standardization processing, and optimizes the data collection strategy and data preprocessing strategy by using the historical performance analysis results of the data.

[0014] Optionally, after executing step S4, the compliance report is manually audited, the AI-Agent optimizes the compliance report according to the manual auditing result, and then the AI-Agent learns the optimized compliance report to improve the accuracy and readability of the subsequent generated compliance report.

[0015] Optionally, after executing step S4, the AI-Agent continuously monitors the entire process of data collection, data preprocessing, and data compliance evaluation, and realizes self-updating through a continuous learning mechanism.

[0016] The self-updated AI-Agent optimizes the data collection strategy and data preprocessing strategy according to the evaluation result to ensure the continuous effectiveness of the compliance detection.

[0017] In a second aspect, the present application provides a data compliance detection system based on a large language model and an AI-Agent, which solves the above technical problems by adopting the following technical solutions:

[0018] A data compliance detection system based on a large language model and an AI-Agent, comprising:

[0019] An optimization processing module for automatically collecting data from multiple data sources by the AI-Agent, performing preprocessing and standardization on the collected data, performing historical performance analysis on the standardized data, and optimizing the data collection strategy and data preprocessing strategy of the AI-Agent based on the analysis results;

[0020] An extraction optimization module for performing feature extraction and feature optimization on the preprocessed data by the LLaMA-2 model;

[0021] An analysis adjustment module for performing deep learning analysis on the optimized features by the LLaMA-2 model, and manually fine-tuning the parameters of the LLaMA-2 model based on the analysis results;

[0022] A rule acquisition module for automatically acquiring the latest compliance rules by a rule engine;

[0023] The calling evaluation module is configured to call the LLaMA-2 model through the AI-Agent, and the LLaMA-2 model performs compliance evaluation on the preprocessed data based on the obtained compliance rules, thereby generating a compliance report.

[0024] Optionally, the optimization processing module involved sequentially performs preprocessing, standardization, and historical performance analysis on the collected data through the AI-Agent, and specifically includes the following steps.

[0025] According to the pre-set cleaning rules, the AI-Agent performs the preprocessing operation of data cleaning on the collected data;

[0026] According to the pre-set standardization rules, the AI-Agent performs format standardization on the cleaned data;

[0027] The AI-Agent performs historical performance analysis on the data after standardization processing, and optimizes the data collection strategy and data preprocessing strategy by using the historical performance analysis result of the data.

[0028] Optionally, after the calling evaluation module involved generates the compliance report, the compliance report is manually audited, the AI-Agent optimizes the compliance report according to the manual auditing result, and then the AI-Agent learns the optimized compliance report to improve the accuracy and readability of the subsequent generated compliance report.

[0029] Optionally, the AI-Agent involved continuously monitors the entire process of data collection, data preprocessing, and data compliance evaluation, and realizes self-updating through a continuous learning mechanism.

[0030] The self-updated AI-Agent then optimizes the data collection strategy and data preprocessing strategy according to the evaluation result, so as to ensure continuous and effective compliance detection.

[0031] The data compliance detection method and system based on the large language model and the AI-Agent have the following beneficial effects compared with the prior art:

[0032] 1. The combination of the large language model and the AI-Agent realizes the automation and high efficiency of data compliance detection, reduces the demand for manual intervention, improves the accuracy and adaptability of data compliance detection, and can cope with changing compliance requirements.

[0033] 2. The AI-Agent has the ability of continuous monitoring and self-updating, ensures real-time compliance detection, and can timely adapt to new compliance rules and changes. BRIEF DESCRIPTION OF DRAWINGS

[0034] ATTACHMENT Figure 1is a flow chart of a method according to embodiment 1 of the present invention;

[0035] Attachment Figure 2 This is a module connection diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the technical solution, the technical problems solved and the technical effects of the present invention more clear, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0037] The embodiment involving AI-Agent and LLaMA-2 model is explained as follows:

[0038] From a macro perspective, an AI-Agent is an intelligent life form capable of making autonomous decisions and executing tasks independently of human control. In the context of LLM, an AI-Agent can be understood as an intelligent entity capable of autonomous perception, planning, decision-making, and executing complex tasks based on a large language model. It can achieve given goals step by step through independent thinking and the use of tools, without requiring human intervention to specify each step. In summary, an agent is comprised of LLM + memory + planning + tool use, with each component being essential to an AI-Agent. A standard AI-Agent typically includes the following capabilities: 1) Perception: It can perceive its environment through sensors or data input. For example, self-driving cars use radar and cameras to perceive their surroundings. 2) Reasoning: It makes decisions based on the information collected, which may involve simple "if-else" logic rules or complex machine learning algorithms. 3) Action: It takes some action to achieve its goal. For example, a nanny robot may return the dinner plates to the kitchen, and a self-driving car may adjust its speed or steering. 4) Learning: It improves its behavior based on experience. 5) Adaptability: It can adapt to changing environments or needs. 6) Interaction: It interacts with humans or other AI-Agents to complete its tasks. 7) Autonomy: It has a certain degree of autonomous decision-making ability without human intervention.

[0039] LLaMA-2 models are a series of pre-trained and fine-tuned large language models (LLMs) developed by Meta (formerly Facebook). The features of LLaMA-2 models include: 1) Parameter size: The parameter size of the model ranges from 7 billion to 70 billion to accommodate different application scenarios and performance requirements. 2) Optimization scenarios: Among them, Llama2-Chat is a model specially optimized for dialogue scenarios, which outperforms open-source dialogue models in multiple benchmark tests and is recognized by humans in terms of usefulness and safety. 3) Training data and context window: The training data set of LLaMA-2 models reaches 2 trillion tokens, and the context length is expanded from Llama's 2048 to 4096, which enables the model to understand and generate longer text. 4) Open source and commercial use: Meta provides open source code for LLaMA-2 models and allows them to be used for research and commercial purposes. Overall, the release of LLaMA-2 models brings new research and application possibilities to the AI field, and their open source and free commercial characteristics enable more developers and enterprises to use this powerful tool for innovation and product development.

[0040] Embodiment one:

[0041] In combination with the accompanying Figure 1 , the embodiment proposes a data compliance detection method based on large language models and AI-Agent, which includes the following steps:

[0042] S1, through AI-Agent, automatically collect data from multiple data sources, perform preprocessing and standardization on the collected data, perform historical performance analysis on the standardized data, and optimize the data collection strategy and data preprocessing strategy of AI-Agent based on the analysis results.

[0043] Perform step S1, AI-Agent performs preprocessing, standardization, and historical performance analysis on the collected data, which specifically includes:

[0044] According to the pre-set cleaning rules, AI-Agent performs data cleaning preprocessing operations on the collected data;

[0045] According to the pre-set standardization rules, AI-Agent performs format standardization on the cleaned data;

[0046] AI-Agent performs historical performance analysis on the standardized data, and optimizes the data collection strategy and data preprocessing strategy using the historical performance analysis results of the data.

[0047] S2, through the LLaMA-2 model, perform feature extraction and feature optimization on the preprocessed data.

[0048] S3, the LLaMA-2 model performs deep learning analysis on the optimized features, and a human fine-tunes the parameters of the LLaMA-2 model based on the analysis results.

[0049] S4, the AI-Agent automatically acquires the latest compliance rules through a rule engine, then calls the LLaMA-2 model, and performs compliance evaluation on the preprocessed data based on the acquired compliance rules, and further generates a compliance report.

[0050] The method described in this embodiment also includes:

[0051] S5, after generating the compliance report, a human audits the compliance report, the AI-Agent optimizes the compliance report according to the human auditing results, then the AI-Agent learns the optimized compliance report to improve the accuracy and readability of subsequent generated compliance reports.

[0052] S6, the AI-Agent continuously monitors the entire process of data collection, data preprocessing, and data compliance evaluation, and realizes self-updating through a continuous learning mechanism; the self-updated AI-Agent optimizes the data collection strategy and the data preprocessing strategy according to the evaluation results to ensure the continuous effectiveness of the compliance detection.

[0053] Embodiment two:

[0054] In combination with the accompanying Figure 2 , this embodiment proposes a data compliance detection system based on a large language model and an AI-Agent, which includes:

[0055] An optimization processing module for automatically collecting data from multiple data sources through an AI-Agent, performing preprocessing and standardization on the collected data, performing historical performance analysis on the standardized data, and optimizing the data collection strategy and the data preprocessing strategy of the AI-Agent based on the analysis results;

[0056] An extraction optimization module for performing feature extraction and feature optimization on the preprocessed data through an LLaMA-2 model;

[0057] An analysis adjustment module for performing deep learning analysis on the optimized features through an LLaMA-2 model, and a human fine-tunes the parameters of the LLaMA-2 model based on the analysis results;

[0058] A rule acquisition module for automatically acquiring the latest compliance rules through a rule engine;

[0059] A calling and evaluation module for calling an LLaMA-2 model through an AI-Agent, the LLaMA-2 model performing compliance evaluation on the preprocessed data based on the acquired compliance rules, and further generating a compliance report.

[0060] In this embodiment, the optimization processing module sequentially performs preprocessing, standardization, and historical performance analysis on the collected data through the AI-Agent, specifically including:

[0061] According to the pre-set cleaning rules, the AI-Agent performs the preprocessing operation of data cleaning on the collected data;

[0062] According to the pre-set standardization rules, the AI-Agent performs format standardization on the cleaned data;

[0063] The AI-Agent performs historical performance analysis on the data after standardization processing, and optimizes the data collection strategy and data preprocessing strategy using the historical performance analysis results of the data.

[0064] In this embodiment, after the evaluation module generates the compliance report, the compliance report is manually audited, the AI-Agent optimizes the compliance report according to the manual audit result, and then the AI-Agent learns the optimized compliance report to improve the accuracy and readability of the subsequent generated compliance report.

[0065] In this embodiment, the AI-Agent continuously monitors the entire process of data collection, data preprocessing, and data compliance evaluation, and realizes self-updating through a continuous learning mechanism; the AI-Agent after self-updating then optimizes the data collection strategy and data preprocessing strategy according to the evaluation result, so as to ensure the continuous effectiveness of the compliance detection.

[0066] As can be seen from the above, the data compliance detection method and system based on the large language model and the AI-Agent can realize the automation and high efficiency of data compliance detection, reduce the demand for manual intervention, improve the accuracy and adaptability of data compliance detection, and can cope with the changing compliance requirements.

[0067] The principles and implementation modes of the present application are described in detail above with reference to specific examples. These examples are only used to help understand the core technical content of the present application. Based on the above specific embodiments of the present application, any improvement and modification of the present application made by those skilled in the art without departing from the principles of the present application shall fall within the scope of the patent protection of the present application.

Claims

1. A data compliance detection method based on a large language model and AI-Agent, characterized in that: The steps include: S1. Automatically collect data from multiple data sources through AI-Agent, preprocess and standardize the collected data, perform historical performance analysis on the standardized data, and optimize the AI-Agent's data collection and data preprocessing strategies based on the analysis results; S2, perform feature extraction and feature optimization on the preprocessed data through the LLaMA-2 model; S3 and LLaMA-2 models perform deep learning analysis on the optimized features, and manually fine-tune the parameters of the LLaMA-2 model based on the analysis results; S4, AI-Agent automatically obtains the latest compliance rules through the rule engine, and then calls the LLaMA-2 model to perform compliance assessment on the pre-processed data based on the obtained compliance rules, and then generates a compliance report.

2. The data compliance detection method based on a large language model and AI-Agent according to claim 1 is characterized in that: In step S1, the AI-Agent performs preprocessing, standardization, and historical performance analysis on the collected data, specifically including: According to the pre-set cleaning rules, AI-Agent performs pre-processing operations for data cleaning on the collected data; According to pre-set standardization rules, AI-Agent performs format standardization on the cleaned data; AI-Agent performs historical performance analysis on the standardized data and uses the historical performance analysis results to optimize data collection and data preprocessing strategies.

3. The data compliance detection method based on a large language model and AI-Agent according to claim 1 is characterized in that: After executing step S4 and generating the compliance report, the compliance report is manually reviewed and the AI-Agent optimizes the compliance report based on the manual review results. Subsequently, the AI-Agent learns the optimized compliance report to improve the accuracy and readability of subsequent compliance reports.

4. The data compliance detection method based on a large language model and AI-Agent according to claim 1 is characterized in that: After executing step S4, the AI-Agent continuously monitors the entire process of data collection, data preprocessing, and data compliance assessment, and achieves self-update through a continuous learning mechanism; The self-updated AI-Agent optimizes data collection and data preprocessing strategies based on the evaluation results to ensure continuous and effective compliance detection.

5. A data compliance detection system based on a large language model and AI-Agent, characterized by: It includes: An optimization processing module is used to automatically collect data from multiple data sources through AI-Agent, perform preprocessing and standardization on the collected data, conduct historical performance analysis on the standardized data, and optimize the AI-Agent's data collection and data preprocessing strategies based on the analysis results; Extraction and optimization module, used to perform feature extraction and feature optimization on preprocessed data through LLaMA-2 model; The analysis and adjustment module is used to perform deep learning analysis on the optimized features through the LLaMA-2 model and manually fine-tune the parameters of the LLaMA-2 model based on the analysis results; The rule acquisition module is used to automatically obtain the latest compliance rules through the rule engine; The evaluation module is called to call the LLaMA-2 model through the AI-Agent. The LLaMA-2 model performs compliance evaluation on the pre-processed data based on the obtained compliance rules and then generates a compliance report.

6. The data compliance detection system based on a large language model and AI-Agent according to claim 5 is characterized in that: The optimization processing module performs preprocessing, standardization, and historical performance analysis on the collected data through AI-Agent, specifically including: According to the pre-set cleaning rules, AI-Agent performs pre-processing operations for data cleaning on the collected data; According to pre-set standardization rules, AI-Agent performs format standardization on the cleaned data; AI-Agent performs historical performance analysis on the standardized data and uses the historical performance analysis results to optimize data collection and data preprocessing strategies.

7. The data compliance detection system based on a large language model and AI-Agent according to claim 5 is characterized in that: After the evaluation module is called to generate a compliance report, the compliance report is manually reviewed, and the AI-Agent optimizes the compliance report based on the manual review results. Subsequently, the AI-Agent learns the optimized compliance report to improve the accuracy and readability of subsequently generated compliance reports.

8. The data compliance detection system based on a large language model and AI-Agent according to claim 5 is characterized in that: The AI-Agent continuously monitors the entire process of data collection, data preprocessing, and data compliance assessment, and achieves self-update through a continuous learning mechanism; The self-updated AI-Agent then optimizes data collection strategies and data preprocessing strategies based on the evaluation results to ensure continuous and effective compliance detection.

Images