Multi-agent-based data analysis report generation method and device and storage medium

Abstract

The invention belongs to the technical field of large models, and discloses a multi-agent-based data analysis report generation method and device and a storage medium, and the method comprises the steps: obtaining metadata information of a target data table, and generating a structured data table containing standardized data items and compliance rules thereof based on a business knowledge base; configuring a corresponding data quality detection rule based on the detection index knowledge base; performing stratified sampling on the data of the structured data table according to the data sensitivity, the time dimension and the business importance by using a data sampling agent; verifying the sampled data set by using a distributed stream processing engine; according to a detection result, generating an intelligent agent output data item-level qualified rate, a problem reason and a change trend; and calling the report template in the data report knowledge base through the detection report generation agent, and displaying the information in a visual chart mode. Therefore, the data quality analysis and report generation method can deeply integrate business knowledge, and automation and intelligentization of the whole process are realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of large models, and particularly relates to a data analysis report generation method and device based on multiple agents and a storage medium. BACKGROUND

[0002] With the deepening of informationization construction, a large data platform converges unprecedentedly large and various internal and external data resources. As the source of the platform, the quality of data, especially the accuracy, completeness, consistency and timeliness, is directly related to the efficiency and reliability of core businesses such as intelligence research and judgment, case investigation and command decision. Therefore, comprehensive and efficient quality analysis and monitoring of converged data have become a key prerequisite for stable operation and value release of the large data platform.

[0003] At present, data quality analysis work mainly relies on manual operation of data operators. This traditional mode usually includes the following links: the operator needs to first deeply understand the complex business logic corresponding to the target data table and accurately master the business meaning of each data item; then, the operator needs to manually check the content and define the rules for the primary key, foreign key, business time and key business field; in addition, the operator needs to identify the dictionary table and basic table related to the data and perform correlation analysis to verify the consistency. The whole process not only requires the operator to have deep business knowledge, but also needs to invest a lot of patience and care, which is a highly intensive labor that depends on expert experience.

[0004] However, this traditional mode dominated by manpower exposes many severe limitations in practice. First, the efficiency is low. In the face of data of the order of billions or even higher, manual analysis is slow, and the cycle from data extraction, rule making to result review is long, which is difficult to meet the high requirements of timeliness of real combat business. Second, the cost is high. In order to ensure the quality of analysis, a large number of high-skilled talents are needed, and a complex multi-person review process is needed, resulting in high costs of manpower, computing power and time. Third, the accuracy is difficult to guarantee. Human operation inevitably has negligence and subjective judgment deviation, and the error probability is high, which may cause missed detection or misjudgment and affect the data reliability. Fourth, it is difficult to deposit and reuse knowledge. The experience and knowledge of experts exist in the individual brain or scattered documents, which is difficult to systematize and standardize for reuse in different projects, resulting in repeated analysis of similar problems and failure to fully realize the value of knowledge assets. Therefore, developing a data quality analysis and report generation method that can deeply integrate business knowledge and realize full-process automation and intelligentization is not only an urgent need to cope with the current challenges of big data management, but also an inevitable trend of technological development. SUMMARY

[0005] In order to realize the full-process automation and intelligentization of the data quality analysis and report generation method, in the first aspect, the application provides a data analysis report generation method based on multiple agents, which comprises the following steps: Obtaining metadata information of a target data table, wherein the metadata information comprises data table structure, data item business semantics and data scale characteristics; Based on a business knowledge base, a data item analysis agent is used to perform natural language understanding and standardization processing on the metadata information, generate a structured data table comprising standardized data items and compliance rules, and update the business knowledge base by using the generated structured data table; Based on a detection index knowledge base, a detection index configuration agent is used to configure corresponding data quality detection rules for the standardized data items in the structured data table, wherein the data quality detection rules comprise integrity, consistency, timeliness, accuracy and rationality rules; Using a data sampling agent, the data of the structured data table is hierarchically sampled according to data sensitivity, time dimension and business importance, a sampling data set is generated, and the sampling data is subjected to security desensitization processing; The sampling data set is verified by using a distributed stream processing engine; a data item level qualified rate is output by a detection result generation agent; the failure mode of the data quality detection rules is determined to identify the problem cause, and the change trend of the data item level qualified rate is identified by comparing historical detection results; A detection report generation agent calls a report template in a data report knowledge base to display the data item level qualified rate, the problem list and the change trend in the form of visual charts.

[0006] In a possible implementation, the detection index configuration agent is used to configure corresponding data quality detection rules for the standardized data items in the structured data table based on the detection index knowledge base, which comprises the following steps: In the case that the standardized data item is an ID number, a detection index configuration agent is used to add a verification formula of national standard GB11643-1999 according to the detection index knowledge base; In the case that the standardized data item is a case occurrence time, a detection index configuration agent is used to add a logical rule that the case occurrence time cannot be later than the current system time according to the detection index knowledge base; In the case that the standardized data item is a telephone number, a detection index configuration agent is used to add a business rule that the null value rate cannot exceed a preset proportion according to the detection index knowledge base; In the case that the standardized data item is an involved amount, if the case type data item is a theft case, a detection index configuration agent is used to add a business rule that the involved amount cannot exceed a preset amount according to the detection index knowledge base.

[0007] In a possible implementation, the data sampling agent is used to hierarchically sample data of the structured data table according to data sensitivity, time dimension and business importance, to generate a sampled data set, comprising: For a sensitive data item of the structured data table, the data sampling agent is used to sample at a first sampling rate; For the structured data table in the past three months, the data sampling agent is used to sample at a second sampling rate, and for the historical structured data table, the data sampling agent is used to sample at a third sampling rate, the second sampling rate being greater than the third sampling rate; For the core structured data table, the data sampling agent is used to sample at a fourth sampling rate.

[0008] In a possible implementation, the sampled data set is verified by using a distributed stream processing engine, comprising: The sampled data set is verified for integrity and consistency by using a Spark SQL stream engine, and timeliness is verified by using a Flink stream engine; The method further comprises: If the sampled data set meets a preset special rule, a flow of detecting legality is triggered by using an associated metadata quality index mapping table.

[0009] In a possible implementation, the problem cause is determined based on the failure mode of the data quality detection rule, comprising: If the null value of the birth date data item occurs in the entry system, it is determined that the problem cause is that the interface timeout causes data truncation; The method further comprises: In combination with the influence of the business knowledge base on quantifying data item errors, a rectification suggestion is generated for a data item with a data item level qualified rate lower than a threshold value.

[0010] In a second aspect, the present application provides a data analysis report generation device based on multiple agents, comprising: An acquisition module is configured to acquire metadata information of a target data table, the metadata information comprising data table structure, data item business semantics and data scale characteristics; An analysis module is configured to use a data item analysis agent to perform natural language understanding and standardization processing on the metadata information based on a business knowledge base, to generate a structured data table comprising standardized data items and compliance rules, and to update the business knowledge base by using the generated structured data table; a rule configuration module, configured to configure a corresponding data quality detection rule for a standardized data item in the structured data table based on a detection index knowledge base by using a detection index configuration agent, wherein the data quality detection rule comprises integrity, consistency, timeliness, accuracy and rationality rules; a sampling module, configured to perform hierarchical sampling on data of the structured data table according to data sensitivity, time dimension and business importance by using a data sampling agent, to generate a sampling data set, and to perform security desensitization processing on the sampling data; a detection result generation module, configured to perform verification on the sampling data set by using a distributed stream processing engine, to output a data item level qualified rate by using a detection result generation agent, to determine a problem cause based on a failure mode of the data quality detection rule, and to identify a change trend of the data item level qualified rate by comparing historical detection results; a detection report generation module, configured to call a report template in a data report knowledge base by using a detection report generation agent, and to display the data item level qualified rate, a problem list and a change trend in a visual chart.

[0011] In a possible implementation, the rule configuration module is specifically configured to: in a case where the standardized data item is an ID number, a verification formula of a national standard GB11643-1999 is added by using the detection index configuration agent according to the detection index knowledge base; in a case where the standardized data item is a case occurrence time, a logical rule that the case occurrence time cannot be later than a current system time is added by using the detection index configuration agent according to the detection index knowledge base; in a case where the standardized data item is a telephone number, a business rule that a null value rate is not more than a preset proportion is added by using the detection index configuration agent according to the detection index knowledge base; in a case where the standardized data item is an involved amount, if a case type data item is a theft case, a business rule that the involved amount is not more than a preset amount is added by using the detection index configuration agent according to the detection index knowledge base.

[0012] In a possible implementation, the sampling module is specifically configured to: for a sensitive data item of the structured data table, sampling is performed at a first sampling rate by using the data sampling agent; for a structured data table in the last three months, sampling is performed at a second sampling rate by using the data sampling agent, for a historical structured data table, sampling is performed at a third sampling rate by using the data sampling agent, and the second sampling rate is greater than the third sampling rate; for a core structured data table, sampling is performed at a fourth sampling rate by using the data sampling agent.

[0013] In a possible implementation, the detection result generation module is specifically configured to: perform integrity and consistency verification on the sampling data set by using a Spark SQL stream engine, and perform timeliness verification on the sampling data set by using a Flink stream engine; The device further includes: An exception processing module is configured to trigger a flow of detecting legality by using an associated metadata quality index mapping table if the sampling data set meets preset special rules.

[0014] In a third aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement any of the above data analysis report generation methods.

[0015] The multi-agent-based data analysis report generation method provided by the embodiments of the present application includes: obtaining metadata information of a target data table, wherein the metadata information includes data table structure, data item business semantics, and data size characteristics; based on a business knowledge base, using a data item analysis agent to perform natural language understanding and standardization processing on the metadata information, generating a structured data table containing standardized data items and compliance rules thereof, and updating the business knowledge base using the generated structured data table; based on a detection index knowledge base, using a detection index configuration agent to configure corresponding data quality detection rules for the standardized data items in the structured data table, wherein the data quality detection rules include integrity, consistency, timeliness, accuracy, and rationality rules; using a data sampling agent to perform hierarchical sampling on the data of the structured data table according to data sensitivity, time dimension, and business importance, generating a sampling data set, and performing security desensitization processing on the sampling data; using a distributed stream processing engine to verify the sampling data set; outputting a data item level qualified rate by using a detection result generation agent; determining a problem cause based on a failure mode of the data quality detection rules, and identifying a change trend of the data item level qualified rate by comparing historical detection results; calling a report template in a data report knowledge base by using a detection report generation agent to display the data item level qualified rate, a problem list, and a change trend in a visual chart manner. The data quality analysis and report generation method can deeply integrate business knowledge and realize full-process automation and intelligentization. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of the multi-agent-based data analysis report generation method provided by the embodiments of the present application is shown in the figure. Figure 2 A logic diagram of the multi-agent-based data analysis report generation method provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0017] The application will be described in detail below through examples.

[0018] With the deepening of informatization construction, the big data platform has gathered unprecedentedly large and diverse internal and external data resources. As the source of the platform, the quality of data, especially its accuracy, completeness, consistency, and timeliness, is directly related to the efficiency and reliability of core businesses such as intelligence research and judgment, case investigation, and command decision. Therefore, comprehensive and efficient quality analysis and monitoring of the gathered data have become a key prerequisite for the stable operation and value release of the big data platform.

[0019] Currently, data quality analysis mainly relies on manual operation by data operators. This traditional mode usually includes the following steps: the operator needs to first deeply understand the complex business logic corresponding to the target data table and accurately grasp the business meaning of each data item; then, the operator needs to manually check the content and define the rules for the primary key, foreign key, business time, and key business fields; in addition, the operator needs to identify the related dictionary table and basic table and perform correlation analysis to verify consistency. The entire process not only requires the operator to have deep business knowledge, but also requires a lot of patience and care, and is a highly intensive labor that relies heavily on expert experience.

[0020] However, this traditional mode dominated by manpower has exposed many serious limitations in practice. First, the efficiency is low. In the face of data of the order of billions or even higher, manual analysis is slow, and the period from data extraction, rule formulation to result review is long, which cannot meet the high requirements of timeliness of real combat business. Second, the cost is high. To ensure analysis quality, a large number of high-skilled talents are needed, and a complex multi-person review process is needed, resulting in high costs of manpower, computing power, and time. Third, the accuracy is difficult to guarantee. Human operation inevitably has negligence and subjective judgment bias, and the error probability is high, which may cause missed detection or misjudgment and affect data reliability. Fourth, knowledge is difficult to deposit and reuse. The experience and knowledge of experts are mostly in the individual's brain or scattered documents, which are difficult to systematize and standardize for reuse in different projects, resulting in repeated analysis of similar problems and failure to fully realize the value of knowledge assets. Therefore, developing a data quality analysis and report generation method that can deeply integrate business knowledge and realize full-process automation and intelligentization is not only an urgent need to cope with the current challenges of big data management, but also an inevitable trend of technological development.

[0021] In recent years, artificial intelligence technology, especially large language models, has made breakthroughs in natural language understanding, logical reasoning, and content generation. This provides a new technical path to solve the above-mentioned difficulties. Based on AI large models, vectorized knowledge bases, and intelligent agent process automation technologies, it is possible to build an intelligent data quality analysis system that can simulate expert thinking and automatically perform complex tasks. This means that we have the opportunity to free data operators from heavy and repetitive manual labor and move to a higher value decision-making supervision link, thereby achieving a qualitative leap in data quality inspection efficiency.

[0022] In a first aspect, see Figure 1 The present application proposes a data analysis report generation method based on multiple agents, which comprises: S101, obtaining metadata information of a target data table, the metadata information comprising data table structure, data item business semantics and data size characteristics.

[0023] Data collection is the cornerstone of data detection, and data table structure, data item business semantics and data size characteristics can be systematically and automatically collected through the metadata management system of the connected business system. The data table structure includes table name, data item name, data type, primary and foreign key constraints, etc. The data item business semantics is the legal meaning of the data items such as household address and involved amount. The data size characteristics include the total table row size such as the billion-level call record table, the partitioned data volume according to year and month, the daily incremental data record of about 2 million, and the update frequency mode. The completeness of this step directly determines the effectiveness of the subsequent detection, and any missing will lead to quality evaluation deviation, so a real-time monitoring mechanism needs to be established to alarm the metadata collection abnormalities.

[0024] S102, based on the business knowledge base, using data item analysis agents to perform natural language understanding and standardization processing on the metadata information, generating a structured data table containing standardized data items and compliance rules, and updating the business knowledge base using the generated structured data table.

[0025] This step converts raw metadata into a standardized data item set that meets industry standards through natural language processing techniques and business knowledge base collaboration. Specifically, a pre-trained BERT model can be used to parse data items, such as mapping "SFZH" to "citizen ID number". For ambiguous terms, such as "address", which can refer to "registered address" or "residential address", an automatic review process is triggered. At the same time, the field type, length, and value range of the data item are checked for compliance, such as the ID number must be 18 characters, and the gender value range is limited to {'male', 'female', 'unknown'}. The output forms a structured standard data item list, which is updated back to the business knowledge base to form a closed-loop optimization mechanism, effectively solving the quality misjudgment problem caused by chaotic naming, and providing authoritative basis for subsequent detection rule configuration.

[0026] S103, based on the detection index knowledge base, using the detection index configuration agent to configure the corresponding data quality detection rules for the standardized data items in the structured data table, including integrity, consistency, timeliness, accuracy and rationality rules.

[0027] In the case of the standardized data item being an ID number, according to the detection index knowledge base, the detection index configuration agent adds the verification formula of national standard GB11643-1999; the standard specifies that the citizen ID number is an 18-digit structure: the first 6 digits are administrative division code, the 7th-14th digits are birth date code, the 15th-17th digits are sequence code, and the 18th digit is verification code; In the case of the standardized data item being the time of the incident, according to the detection index knowledge base, the detection index configuration agent adds the logical rule that it cannot be later than the current system time; In the case of the standardized data item being a phone number, according to the detection index knowledge base, the detection index configuration agent adds the business rule that the null value rate does not exceed the pre-set proportion; such as reducing the "phone number null value rate" tolerance threshold from 5% to 1%; In the case of the standardized data item being the amount involved, if the case type data item is a theft case, according to the detection index knowledge base, the detection index configuration agent adds the business rule that the amount involved does not exceed the pre-set amount.

[0028] Finally, output the rule set in JSON-Schema format, each rule containing a unique ID, applicable data item, detection logic, threshold value and repair suggestion, ensuring that the detection system meets industry general standards and accurately adapts to frontline operational needs.

[0029] S104, using the data sampling agent, according to the data sensitivity, time dimension and business importance, the data of the structured data table is stratified sampled, the sampling data set is generated, and the sampling data is safely desensitized.

[0030] The design of the sampling scheme directly affects the credibility and efficiency of the detection results. The data sampling agent is designed based on the data size characteristics and security requirements as follows: For sensitive data items of the structured data table, the data sampling agent is used to sample at a first sampling rate; for example, the sampling rate of personal sensitive information fields such as ID numbers is controlled within 15%; For the structured data table in the past three months, the data sampling agent is used to sample at a second sampling rate, and for the historical structured data table, the data sampling agent is used to sample at a third sampling rate, the second sampling rate being greater than the third sampling rate; for example, the sampling rate of the data in the past three months is 10%, and the sampling rate of the historical data is reduced to 3%; For the core structured data table, the data sampling agent is used to sample at a fourth sampling rate; for example, the sampling rate of the core data table is increased to 20%.

[0031] At the same time, strict security compliance processing is implemented, and instant desensitization is performed on the sampled data, such as keeping only the first 6 digits of the ID number and replacing the other digits with asterisks, and transmitting through encryption. The generated sampling script supports Spark SQL, Flink, and other stream engines for execution, and records the random seed value to ensure the reproducibility of the results.

[0032] S105, using a distributed stream processing engine to verify the sampled data set; generating an intelligent agent output data item level qualified rate through the detection result; determining the problem cause based on the failure mode of the data quality detection rule, and identifying the change trend of the data item level qualified rate by comparing the historical detection results.

[0033] The Spark SQL stream engine is used to verify the integrity (such as null check) and consistency (such as regular matching) of the sampled data set, and the RDD (Resilient Distributed Dataset) parallel computing is used to achieve a daily throughput of PB level; the Flink stream engine is used to verify the timeliness of the sampled data set, such as the call time not lagging behind the current time.

[0034] If the sampled data set meets the preset special rules, the associated metadata quality index mapping table is triggered to detect the legality process.

[0035] For example, the Hong Kong and Macao resident ID number does not meet the mainland verification rules, in which case the exception process is automatically triggered, and the associated metadata quality index mapping table is used to detect the legality of the Hong Kong and Macao resident ID number. The metadata quality index mapping table is used to point to the Hong Kong and Macao ID number rule library, and the engine finds the verification logic in the Hong Kong and Macao ID number rule library and then performs detection, successfully avoiding the misjudgment caused by the Hong Kong and Macao resident ID number not meeting the mainland verification rules.

[0036] The full-link tracking system records the detection path through a unique task ID, covering elements such as rule version, sampling parameters, execution node, and timestamp to support audit traceability. The detection results are stored in partitioned Parquet files, only retaining pass / fail identification, failure rule ID, and original value hash value, to eliminate the risk of sensitive information leakage.

[0037] One example of determining the cause of the problem based on the failure mode of the data quality detection rule can be: If the null value of the birth date data item occurs in the entry system, it is determined that the cause of the problem is that the interface timeout causes data truncation; In combination with the quantification of the impact of data item errors based on the business knowledge base, rectification suggestions are generated for data items with a data item level qualification rate below a threshold.

[0038] The detection result generating agent analyzes the mode of the failed data. For example, it is found that the null value of the "birth date" field comes from the data entered by a certain city branch A in the past week. Then, it associates the data blood relationship and metadata, and finds that these data are all synchronized from a certain business system C through interface B. Based on this mode, a highly suspicious root cause hypothesis can be generated: "interface B" of "certain business system C" may have recently experienced unstable performance or timeout, resulting in truncation during data synchronization and loss of "birth date" information. In this way, the problem is accurately located from the data level to the business system or operation process level, providing a clear direction for repair and avoiding mutual shirking of each department.

[0039] In combination with the quantification of the impact of the problem based on the business knowledge base, such as the amount of money error that may cause 20% of the case analysis to fail.

[0040] Intelligent rectification suggestions are generated for high-frequency problems, such as adding a province abbreviation whitelist check for the "license plate number" field.

[0041] In view of the above root cause of interface timeout leading to data truncation, the suggestions may include: notifying "certain city branch A" to suspend data entry through "interface B" and change to manual supplementary verification. Or contact the operation and maintenance team of "certain business system C" to check and optimize the performance of "interface B" and increase the timeout retry mechanism. Or set up real-time monitoring rules for data integrity for "interface B" in the data access platform, and alarm immediately in the future if the null value rate exceeds the threshold.

[0042] The output structured diagnostic report includes three levels of problem classification (serious / warning / reminder), impact range assessment, root cause analysis, and urgent repair scheme, providing scientific support for data governance decision-making.

[0043] S106, the data reporting generation agent calls the reporting template in the data reporting knowledge base to display the data item level qualification rate, problem list, and change trend in a visual chart.

[0044] The data item level qualification rate shows the qualification of each data item, the problem list shows the problems that cause the data item to be unqualified, and the change trend shows the overall qualification rate of the data item.

[0045] Referring to Figure 2 A multi-agent-based data analysis report generation logic diagram is provided for the present application. Data table, data item information, and data size information are first converted into a standard data item set, then data item quality detection indicators are configured, a production data sampling scheme is executed, data sampling detection is performed, and finally data detection result clarification generation and data detection report generation are performed. In this process, the quality detection knowledge base is iterated, the data table and data item description are used to update the business knowledge base, the detection indicator library is used to update the detection indicator knowledge base, the metadata quality indicator mapping table is used to update the metadata quality indicator mapping knowledge base, and the historical detection result is used to update the data reporting knowledge base. The data item analysis agent, the detection indicator configuration agent, the data sampling agent, the detection indicator and detection scheme review agent, the detection result generation agent, and the detection report generation agent complete the data analysis report generation by calling the aforementioned knowledge bases.

[0046] The data analysis report generation system is not a one-time, static tool, but an organic life form that can learn from practice and become smarter with use. It builds a bidirectional enhanced intelligent closed loop between multi-agents and four knowledge bases. This collaborative mechanism is mainly composed of two directions of flow: When the agent detects data, it may encounter a new problem that has never been seen before. For example, a new type of virtual currency fake account appears, whose data pattern (such as specific transaction amount, frequency) is different from the traditional one. The agent can use its reasoning ability to try to configure new detection rules or optimize existing rules for this new problem. These new rules will not take effect directly, but will be submitted to data operation personnel for review, correction and approval. Once approved, this new rule pattern, its corresponding data features and solutions will be structured and deposited into the corresponding knowledge base, such as updating the detection indicator knowledge base.

[0047] In the data detection result clear generation stage, the system generates a large number of high-quality cases. These cases contain the complete logical chain of "problem phenomenon -> root cause analysis -> solution". These high-quality, practice-proven cases are used as training data to continuously fine-tune and optimize the agent. By continuously learning these cases, the agent can more accurately recommend the most appropriate detection rules when it encounters similar data problems in the future. And it can more quickly associate metadata to pinpoint the root cause of the problem. The generated rectification suggestions will be more in line with actual business scenarios.

[0048] This coordination mechanism cannot be separated from the support of the four knowledge bases, which have their own division of labor: 1. Business knowledge base: stores business semantics and data metadata standards. Ensure that the system understands the business.

[0049] 2. Detection index knowledge base: stores various quality rules and indicators. It is the system's detection standard manual.

[0050] 3. Data report knowledge base: stores report templates and historical cases. It is the system's writing material library and best practice library.

[0051] 4. Metadata quality index mapping knowledge base: as a rule routing.

[0052] Secondly, the application provides a data analysis report generation device based on multiple agents, which comprises: An acquisition module is configured to acquire metadata information of a target data table, wherein the metadata information comprises data table structure, data item business semantics, and data size characteristics; An analysis module is configured to perform natural language understanding and standardization processing on the metadata information by using a data item analysis agent based on a business knowledge base, to generate a structured data table comprising standardized data items and compliance rules thereof, and to update the business knowledge base by using the generated structured data table; A rule configuration module is configured to configure corresponding data quality detection rules for the standardized data items in the structured data table by using a detection index configuration agent based on a detection index knowledge base, wherein the data quality detection rules comprise integrity, consistency, timeliness, accuracy, and rationality rules; A sampling module is configured to perform hierarchical sampling on data of the structured data table according to data sensitivity, time dimension, and business importance by using a data sampling agent, to generate a sampling data set, and to perform security desensitization processing on the sampling data; The detection result generation module is configured to check the sampling data set by using a distributed stream processing engine, output a data item level qualified rate by a detection result generation agent, determine a problem cause based on a failure mode of the data quality detection rule, and identify a change trend of the data item level qualified rate by comparing historical detection results. The detection report generation module is configured to call a report template in a data report knowledge base by a detection report generation agent, and display the data item level qualified rate, a problem list, and the change trend in a visual chart.

[0053] In a possible implementation, the rule configuration module is specifically configured to: In a case where the standardized data item is an ID number, a detection index configuration agent is used to add a verification formula of a national standard GB11643-1999 according to a detection index knowledge base; In a case where the standardized data item is a case occurrence time, a detection index configuration agent is used to add a logical rule that the case occurrence time cannot be later than a current system time according to a detection index knowledge base; In a case where the standardized data item is a telephone number, a detection index configuration agent is used to add a business rule that a null value rate is not more than a preset proportion according to a detection index knowledge base; In a case where the standardized data item is an involved amount, if a case type data item is a theft case, a detection index configuration agent is used to add a business rule that the involved amount is not more than a preset amount according to a detection index knowledge base.

[0054] In a possible implementation, the sampling module is specifically configured to: For a sensitive data item of a structured data table, a data sampling agent is used to sample at a first sampling rate; For a structured data table in the last three months, a data sampling agent is used to sample at a second sampling rate, and for a historical structured data table, a data sampling agent is used to sample at a third sampling rate, the second sampling rate being greater than the third sampling rate; For a core structured data table, a data sampling agent is used to sample at a fourth sampling rate.

[0055] In a possible implementation, the detection result generation module is specifically configured to: A Spark SQL stream engine is used to check the sampling data set for integrity and consistency, and a Flink stream engine is used to check the sampling data set for timeliness; The apparatus further includes: The exception processing module is configured to trigger a process of detecting legality by using an associated metadata quality index mapping table if the sampling data set meets a preset special rule.

[0056] In a third aspect, the present application provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the computer program, when executed by a processor, implements any of the above data analysis report generation methods.

[0057] In the above embodiments, the implementation can be wholly or partially achieved by software, hardware, firmware, or any combination thereof. When implemented by software, the implementation can be wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer instructions wholly or partially generate the processes or functions described in the embodiments of the present application. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium, or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, Solid State Disk (SSD)), etc.

[0058] It should be noted that, in this document, the terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0059] The various embodiments in the specification are described in a related manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0060] Although the embodiments of the present application have been shown and described above, it should be understood by those skilled in the art that the above embodiments are exemplary and cannot be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the principles and spirits of the present application within the scope of the present application.

Claims

1. A multi-agent based data analysis report generation method, characterized by, The method comprises: obtaining metadata information of a target data table, the metadata information comprising data table structure, data item business semantics and data scale characteristics; based on a business knowledge base, using a data item analysis intelligent agent to perform natural language understanding and standardization processing on the metadata information, generating a structured data table comprising standardized data items and compliance rules thereof, and updating the business knowledge base using the generated structured data table; based on a detection index knowledge base, using a detection index configuration intelligent agent to configure corresponding data quality detection rules for the standardized data items in the structured data table, the data quality detection rules comprising integrity, consistency, timeliness, accuracy and rationality rules; using a data sampling intelligent agent, performing hierarchical sampling on the data of the structured data table according to data sensitivity, time dimension and business importance, generating a sampling data set, and performing security desensitization processing on the sampling data; using a distributed stream processing engine to verify the sampling data set; an intelligent agent outputs a data item level qualification rate through the detection result; based on the failure mode of the data quality detection rules, the problem cause is determined, and the change trend of the data item level qualification rate is identified by comparing historical detection results; a detection report generation intelligent agent calls a report template in a data report knowledge base to display the data item level qualification rate, the problem list and the change trend in the form of visual charts.

2. The method of claim 1, wherein, The detection index configuration intelligent agent configures corresponding data quality detection rules for the standardized data items in the structured data table based on the detection index knowledge base, comprising: in the case of the standardized data item being an ID number, the detection index configuration intelligent agent adds the verification formula of national standard GB11643-1999 according to the detection index knowledge base; in the case of the standardized data item being a case occurrence time, the detection index configuration intelligent agent adds the logical rule that it cannot be later than the current system time according to the detection index knowledge base; in the case of the standardized data item being a telephone number, the detection index configuration intelligent agent adds the business rule that the null rate does not exceed a preset proportion according to the detection index knowledge base; in the case of the standardized data item being an involved amount, if the case type data item is a theft case, the detection index configuration intelligent agent adds the business rule that the involved amount does not exceed a preset amount according to the detection index knowledge base.

3. The method of claim 1, wherein, The data sampling intelligent agent performs hierarchical sampling on the data of the structured data table according to data sensitivity, time dimension and business importance to generate a sampling data set, comprising: for sensitive data items of the structured data table, the data sampling intelligent agent samples according to a first sampling rate; for structured data tables in the past three months, the data sampling intelligent agent samples according to a second sampling rate, and for historical structured data tables, the data sampling intelligent agent samples according to a third sampling rate, the second sampling rate being greater than the third sampling rate; for core structured data tables, the data sampling intelligent agent samples according to a fourth sampling rate.

4. The method of claim 1, wherein, The distributed stream processing engine verifies the sampling data set, comprising: The sampled data set is checked for integrity and consistency by using a Spark SQL stream engine, and checked for timeliness by using a Flink stream engine; The method further comprises: If the sampled data set meets preset special rules, triggering a correlation metadata quality index mapping table to detect legitimacy.

5. The method of claim 1, wherein, The failure mode based on the data quality detection rule determines the problem cause, including: If the null value of the birth date data item occurs in the entry system, it is determined that the problem cause is data truncation caused by interface timeout; The method further comprises: In combination with the influence of the business knowledge base on quantifying data item errors, generating rectification suggestions for data items with a data item level qualification rate below a threshold.

6. A multi-agent based data analysis report generation apparatus, characterized by, The device comprises: An acquisition module configured to acquire metadata information of a target data table, the metadata information comprising data table structure, data item business semantics, and data size characteristics; An analysis module configured to, based on a business knowledge base, perform natural language understanding and standardization processing on the metadata information by using a data item analysis intelligent agent, generate a structured data table comprising standardized data items and compliance rules thereof, and update the business knowledge base by using the generated structured data table; A rule configuration module configured to, based on a detection index knowledge base, configure corresponding data quality detection rules for the standardized data items in the structured data table by using a detection index configuration intelligent agent, the data quality detection rules comprising integrity, consistency, timeliness, accuracy, and rationality rules; A sampling module configured to perform hierarchical sampling on data of the structured data table according to data sensitivity, time dimension, and business importance by using a data sampling intelligent agent, generate a sampled data set, and perform secure desensitization processing on the sampled data; A detection result generation module configured to check the sampled data set by using a distributed stream processing engine, output a data item level qualification rate by using a detection result generation intelligent agent, determine a problem cause based on a failure mode of the data quality detection rule, and identify a change trend of the data item level qualification rate by comparing historical detection results; A detection report generation module configured to call a report template in a data report knowledge base by using a detection report generation intelligent agent, and display the data item level qualification rate, a problem list, and a change trend in a visual chart.

7. The apparatus of claim 6, wherein, The rule configuration module is specifically configured to: In a case where the standardized data item is an ID number, adding a verification formula of the national standard GB11643-1999 according to the detection index knowledge base by using the detection index configuration intelligent agent; In a case where the standardized data item is a case occurrence time, adding a logical rule that the case occurrence time cannot be later than the current system time according to the detection index knowledge base by using the detection index configuration intelligent agent; In a case where the standardized data item is a telephone number, adding a business rule that the null value rate of the telephone number cannot exceed a preset proportion according to the detection index knowledge base by using the detection index configuration intelligent agent; In a case where the standardized data item is an involved amount, if a case type data item is a theft case, adding a business rule that the involved amount cannot exceed a preset amount according to the detection index knowledge base by using the detection index configuration intelligent agent.

8. The apparatus of claim 6, wherein, The sampling module is specifically used for: For sensitive data items of the structured data table, sampling is performed by the data sampling agent according to a first sampling rate; For the structured data table in the past three months, sampling is performed by the data sampling agent according to a second sampling rate, and for the historical structured data table, sampling is performed by the data sampling agent according to a third sampling rate, the second sampling rate being greater than the third sampling rate; For the core structured data table, sampling is performed by the data sampling agent according to a fourth sampling rate.

9. The apparatus of claim 6, wherein, The detection result generation module is specifically used for: Integrity and consistency verification is performed on the sampling data set by using a Spark SQL stream engine, and timeliness verification is performed on the sampling data set by using a Flink stream engine; The device further comprises: An exception processing module is configured to trigger a legality detection process of an associated metadata quality index mapping table if the sampling data set meets a preset special rule.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-5.

Images