Intelligent agent-based data consistency auditing system

Abstract

The invention relates to the technical field of big data processing, in particular to an agent-based data consistency auditing system, comprising: an agent construction module for constructing an acquisition agent for a data source, the acquisition agent extracting to-be-checked data from the data source; the feature extraction module is used for extracting data features of the to-be-checked data; and the consistency calculation module is used for calculating the data consistency according to the data characteristics. In order to solve the problem that a data consistency checking system in the prior art generally only aims at a single data source and is poor in collection effect, in the embodiment, an agent construction module is introduced, data to be checked are extracted for data of multiple sources respectively, then data features are extracted, and the consistency is calculated; therefore, verification with good consistency on data of different data sources and different data types is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a data consistency audit system based on an intelligent agent. Background Art

[0002] During the operation and management of the big data platform system of the district big data center, the system needs to synchronize and store key data from multiple sources, including the business systems of the district party committee and bureau, the data exchange platform system of the municipal big data center, the business system of the district big data center, etc. The diverse sources of these data lead to significant differences in data standards, collection methods and processing procedures, which may cause the same data to show inconsistency in different systems. This inconsistency poses a major challenge to the data management and decision support of the district big data center.

[0003] In the prior art, there are technical solutions for data consistency checking. For example, Chinese patent CN202210023332.6 discloses a data consistency checking method. For the two subsystems included in the passenger service system: the first data system and the second data system, when the first data system needs to perform a data consistency check, a data check request can be sent to the inspection system, and the data check request includes the data category and the data to be checked (the first data). The inspection system obtains the second data that matches the data category from the second data system according to the data category, and performs a data consistency check using the first data and the second data to obtain the inspection result. That is, the inspection system is deployed in the passenger service system. When the first data system needs to perform a data consistency check, it can actively send a data check request to the inspection system to trigger the inspection system to perform a data consistency check, thereby improving the efficiency and accuracy of the inspection.

[0004] However, during the actual implementation process, the inventors found that this type of technical solution is usually designed for fixed types of data sources with fixed verification conditions, which makes it relatively cumbersome to use and cannot adapt well to multiple data sources. Summary of the invention

[0005] In view of the above problems existing in the prior art, a data consistency audit system based on an intelligent agent is now provided.

[0006] The specific technical solutions are as follows:

[0007] A data consistency audit system based on an intelligent agent, comprising:

[0008] An agent building module, wherein the agent building module builds a collection agent for a data source, and the collection agent extracts data to be checked from the data source;

[0009] A feature extraction module, the feature extraction module is connected to the agent construction module, and the feature extraction module extracts data features from the data to be checked;

[0010] A consistency calculation module, wherein the consistency calculation module is connected to the feature extraction module, and the consistency calculation module calculates data consistency according to the data features.

[0011] On the other hand, the agent building module includes:

[0012] A data source sniffing module, wherein the data source sniffing module collects data interface information from the data source;

[0013] A connection establishment module, the connection establishment module is connected to the data source sniffing module, and the connection establishment module determines a connection protocol according to the interface information;

[0014] A monitoring module, wherein the monitoring module is connected to the connection establishment module, and the monitoring module obtains data changes in the data source, and obtains the data to be checked via the connection protocol when data changes are triggered.

[0015] On the other hand, the feature extraction module comprises:

[0016] A data type determination module, wherein the data type determination module determines a data type based on the data feature;

[0017] A numerical data processing module, the numerical data processing module is connected to the data type determination module, and the numerical data processing module extracts the data features from the numerical data;

[0018] A text data processing module, the text data processing module is connected to the data type determination module, and the text data processing module extracts the data features from the text data;

[0019] A time data processing module, the time data processing module is connected to the data type judgment module, and the time data processing module extracts the data features from the time data.

[0020] On the other hand, the numerical data processing module includes:

[0021] A set mean calculation module, wherein the set mean calculation module calculates a set mean for the numerical data;

[0022] A data feature calculation module, wherein the data feature calculation module is connected to the set mean calculation module, and the data feature calculation module calculates the standard deviation, skewness and kurtosis according to the set mean as the data feature.

[0023] On the other hand, the text data processing module includes:

[0024] A word segmentation module, wherein the word segmentation module performs word segmentation on the text data to obtain a word segmentation result;

[0025] A word frequency statistics module is connected to the word segmentation module.

[0026] On the other hand, the consistency calculation module includes:

[0027] A consistency matrix construction module, wherein the consistency matrix construction module calculates the consistency of the classification data respectively according to the data features;

[0028] An overall consistency calculation module, wherein the overall consistency calculation module is connected to the consistency matrix construction module, and the overall consistency calculation module calculates the data consistency according to the classification data consistency.

[0029] In another aspect, the consistency matrix building module comprises:

[0030] A numerical consistency matrix construction module, wherein the numerical consistency matrix construction module constructs a data consistency matrix for the data features of numerical types and calculates the classification data consistency;

[0031] A text consistency matrix construction module, wherein the text consistency matrix construction module constructs a data consistency matrix for the data features of the text type and calculates the classification data consistency;

[0032] A time consistency matrix construction module is used to construct a data consistency matrix for the data features of the time type and calculate the classification data consistency.

[0033] On the other hand, the overall consistency calculation module performs weighted calculation on the classification data consistency to obtain the data consistency.

[0034] On the other hand, it also includes:

[0035] An execution module, wherein the execution module is connected to the consistency calculation module, and the execution module calls a corresponding processing strategy according to the data consistency and the data type of the data source.

[0036] The above technical solution has the following advantages or beneficial effects:

[0037] In view of the problem that the data consistency verification system in the prior art usually only targets a single data source and has poor collection effect, in this embodiment, an intelligent agent construction module is introduced to extract the data to be verified from multiple sources respectively, and then extract the data features to calculate its consistency, thereby achieving better verification of data consistency for different data sources and different data types. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The embodiments of the present invention will be described more fully with reference to the attached drawings, which are provided for illustration and description only and are not intended to limit the scope of the present invention.

[0039] Figure 1 It is an overall schematic diagram of an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of an intelligent agent building module in an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of a feature extraction module in an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of a numerical data processing module in an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of a text data processing module in an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of a consistency calculation module in an embodiment of the present invention;

[0045] Figure 7 This is a schematic diagram of a consistency matrix construction module in an embodiment of the present invention;

[0046] Figure 8 This is a schematic diagram of an execution module in an embodiment of the present invention; DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0049] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0050] The present invention comprises:

[0051] An agent-based data consistency audit system, such as Figure 1 As shown, including:

[0052] An intelligent agent building module 1, the intelligent agent building module 1 builds a collection intelligent agent for the data source, and the collection intelligent agent extracts the data to be checked from the data source;

[0053] Feature extraction module 2, the feature extraction module 2 is connected to the intelligent agent construction module 1, and the feature extraction module 2 extracts data features from the data to be checked;

[0054] The consistency calculation module 3 is connected to the feature extraction module 2, and the consistency calculation module 3 calculates data consistency according to data features.

[0055] Specifically, in view of the problem that the data consistency verification system in the prior art usually only targets a single data source and has poor collection effect, in this embodiment, an intelligent agent construction module 1 is introduced, and a collection agent is constructed for data from multiple sources for automatic data extraction. The intelligent agent (AI Agent) is a "system driven by a large language model as the brain, with the ability to autonomously understand, perceive, plan, remember and use tools, and can automatically execute and complete complex tasks. The data system has better adaptability to different types of data sources through the collection agent, so as to capture the corresponding data to be checked. Then, the feature extraction module 2 extracts data features for the data to be checked according to the data type of the data to be checked, and the consistency calculation module 3 calculates the data consistency according to the data features, thereby realizing the quantitative representation of data consistency and facilitating subsequent processing.

[0056] In one embodiment, Figure 2 As shown, the intelligent agent building module 1 includes:

[0057] A data source sniffing module 11, which collects data interface information from a data source;

[0058] A connection establishment module 12, the connection establishment module 12 is connected to the data source sniffing module 11, and the connection establishment module 12 determines the connection protocol according to the interface information;

[0059] The monitoring module 13 is connected to the connection establishment module 12. The monitoring module 13 obtains data changes in the data source, and obtains the data to be checked via the connection protocol when the data change is triggered.

[0060] Specifically, in order to achieve a better extraction effect, in this embodiment, an agent with the above functional modules is set in the agent construction module 1. When extracting data from each data source, a thread of a collection agent is created respectively. The collection agent mainly includes the above modules. Among them:

[0061] The data source sniffing module 11 collects data interface information from the data source, mainly including sniffing various interfaces of the data source through an automated script, and inputting the parsed API interface into a large language model to determine the interface that can be used to capture data;

[0062] Then, the connection establishment module 12 constructs a connection protocol corresponding to the interface according to the fields, text content and supported connection modes contained in the interface information;

[0063] Finally, the monitoring module 13 obtains data changes in the data source by initiating a subscription to the data source. After the data source triggers a certain amount of data update, the monitoring module 13 is aware of it through push subscription, and obtains the data to be verified according to the pre-built connection protocol when the data change is triggered.

[0064] The above settings realize the automatic capture process of the data to be verified in various data sources.

[0065] In one embodiment, Figure 3 As shown, the feature extraction module 2 includes:

[0066] A data type determination module 21, the data type determination module 21 determines the data type based on the data features;

[0067] A numerical data processing module 22, the numerical data processing module 22 is connected to the data type judgment module 21, and the numerical data processing module 22 extracts data features from the numerical data;

[0068] A text data processing module 23, the text data processing module 23 is connected to the data type determination module 21, and the text data processing module 23 extracts data features from the text data;

[0069] The time data processing module 24 is connected to the data type judgment module 21, and the time data processing module 24 extracts data features from the time type data.

[0070] Specifically, in order to achieve better data processing effects, in this embodiment, for different types of data sources, the data type is first determined by the data type judgment module 21 for the data to be checked, thereby determining the data type output in different data sources, and then the numerical data processing module 22 extracts data features from the numerical data, the text data processing module 23 extracts data features from the text data, and the time data processing module 24 extracts data features from the time data, thereby achieving effective processing of data from different types of data sources.

[0071] In one embodiment, Figure 4 As shown, the numerical data processing module 22 includes:

[0072] A set mean calculation module 221, the set mean calculation module 221 calculates the set mean of numerical data;

[0073] The data feature calculation module 222 is connected to the set mean calculation module 221, and the data feature calculation module 222 calculates the standard deviation, skewness and kurtosis as data features according to the set mean.

[0074] Specifically, in order to achieve a better processing effect on numerical data, in this embodiment, the above-mentioned numerical data processing module 22 is constructed. Specifically, taking data sources A, B, and C as an example, the numerical values ​​in each data source are calculated respectively. Assume that the numerical data sets collected from the three data sources are A={a 1 ,a 2 ,…,a n},B={b 1 ,b 2 ,…,b n}, C={c 1 ,c 2 ,…,c n}. First, calculate the set mean of each data set Then calculate the standard deviation The extracted features also include skewness and kurtosis. The calculation process of skewness is:

[0075] The formula for calculating kurtosis is:

[0076] In one embodiment, Figure 5 As shown, the text data processing module 23 includes:

[0077] A word segmentation module 231, which performs word segmentation on text data to obtain a word segmentation result;

[0078] A word frequency statistics module 232, which is connected to the word segmentation module 231, and constructs a word frequency vector according to the word segmentation result;

[0079] The word frequency analysis module 233 is connected to the word frequency statistics module 232, and the word frequency analysis module obtains data features according to word frequency vector analysis.

[0080] Specifically, in order to achieve a better extraction effect, this embodiment uses natural language processing technology and vector space model to extract features for text data. Specifically, let the text data set be T = {t 1 ,t 2 ,…,t q}, first the word segmentation module 231 performs word segmentation processing, then the word frequency statistics module 232 counts the frequency of each word in the text, and then constructs a word frequency vector based on the word frequency. i , whose word frequency vector is v i =(f i1 ,f i2 ,…,f ir ), where f ij Indicates the jth word in text t i The frequency of occurrence in is r, and r is the size of the vocabulary. Finally, the word frequency analysis module 233 uses the principal component analysis (PCA) method to reduce the dimension of the constructed word frequency vector, extracts more significant data features and outputs them.

[0081] And, in the time data processing module 24, the time series analysis method is combined to extract features. Assume that the time series data set is S = {s 1 ,s 2 ,…,s t}, where s i =(t i ,y i ), t i is the time point, y i is the corresponding data value. The trend, seasonality, periodicity and other characteristics of the time series can be calculated. For example, the linear regression method is used to fit the trend line of the time series and calculate the trend coefficient. For seasonal characteristics, the seasonal index method can be used for analysis.

[0082] In one embodiment, Figure 6 As shown, the consistency calculation module 3 includes:

[0083] A consistency matrix building module 31, the consistency matrix building module 31 calculates the consistency of the classification data according to the data features;

[0084] The overall consistency calculation module 32 is connected to the consistency matrix construction module 31, and the overall consistency calculation module 32 calculates the data consistency according to the classification data consistency.

[0085] Specifically, in order to achieve better calculation results for the consistency of data from different types of data sources, in this embodiment, first, corresponding data consistency matrices are constructed for the data features obtained from processing multiple data sources, and then the classified data consistency is calculated for the data consistency matrix. Finally, all types of data are summarized, and the overall consistency calculation module 32 calculates the data consistency based on the classified data consistency, thereby achieving the measurement of the consistency of multi-source data.

[0086] In one embodiment, Figure 7 As shown, the consistency matrix building module 31 includes:

[0087] A numerical consistency matrix construction module 311, the numerical consistency matrix construction module 311 constructs a data consistency matrix for data features of numerical types and calculates the classification data consistency;

[0088] A text consistency matrix construction module 312, the text consistency matrix construction module 312 constructs a data consistency matrix for data features of the text type and calculates the classification data consistency;

[0089] The time consistency matrix construction module 313 constructs a data consistency matrix for the data features of the time type and calculates the classification data consistency.

[0090] Specifically, for numerical data, the numerical consistency matrix construction module 311 first constructs the numerical consistency matrix M num . Let M num (i,j) represents the consistency degree of numerical data in system i and system j.

[0091] The calculation formula is:

[0092] M num (i, j) = 1 / (1 + |μ i -μ j |+|σ i -σ j |+|Skewness i -Skewness j |+|Kurtosis i -Kurtosis j |);

[0093] where i, j∈{A, B, C};

[0094] For text data, the text consistency matrix construction module 312 constructs a text consistency matrix M text . Use cosine similarity and other methods to calculate the similarity between texts. Let M text (i,j) represents the consistency of text data in data source i and data source j, and the calculation formula is:

[0095]

[0096] where v ik and v jk are the vector representations of the kth text in data source i and data source j respectively.

[0097] For time-type data, the time consistency matrix construction module 313 constructs the time consistency matrix M time . Dynamic time warping (DTW) and other methods are used to calculate the similarity between time series. Let M time (i, j) represents the consistency of the temporal data in data source i and data source j.

[0098] In one embodiment, the overall consistency calculation module 32 performs weighted calculation on the classification data consistency to obtain data consistency.

[0099] Specifically, the overall consistency index is defined as:

[0101] Among them, α, β, and γ are weight coefficients, which are adjusted according to the actual situation. When C is close to 1, it means that the data tables in the three data sources have high consistency; when C is close to 0, it means that the data tables in the three data sources have low consistency.

[0102] In one embodiment, Figure 8 As shown, it also includes:

[0103] Execution module 4, execution module 4 is connected to consistency calculation module 3, and execution module 4 calls corresponding processing strategy according to data consistency and data type of data source.

[0104] Specifically, in order to achieve better maintainability of data, in this embodiment, an execution module 4 is also constructed in the system, and different processing strategies are configured around different types of data sources. After calculating the data consistency, the execution module 4 compares the triggering conditions of each processing strategy, thereby selecting the triggered processing strategy for execution. For example, when the consistency index C is lower than a certain threshold, an alarm is issued to notify the data administrator of the big data center to conduct data verification and repair. For situations where the difference in numerical data is large, the cause of the difference can be further analyzed, such as whether there is a failure in the data acquisition equipment, system calculation errors, etc. For text data differences, it is possible to check whether there are problems in the data cleaning and standardization process, especially for the problems of irregular addresses and inconsistent business descriptions that often occur in the big data platform. For time data differences, it is possible to check whether the system clock is synchronized and whether the data acquisition timestamp is accurate. The purpose of this module is to ensure the consistency of the data on the district big data platform and improve the data quality.

[0105] Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0106] Those skilled in the art will appreciate that various aspects of the present invention, or possible implementations of various aspects, may be specifically implemented as systems, methods, or computer program products. Therefore, various aspects of the present invention, or possible implementations of various aspects, may take the form of complete hardware embodiments, complete software embodiments (including firmware, resident software, etc.), or embodiments of combined software and hardware aspects, all collectively referred to herein as "circuits," "modules," or "systems." In addition, various aspects of the present invention, or possible implementations of various aspects, may take the form of computer program products, which refer to computer instructions stored in a memory.

[0107] The memory may be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable read-only memory (CD-ROM).

[0108] The processor in the computer reads the computer instructions stored in the memory, so that the processor can execute the functional actions specified in each step or the combination of steps in the flowchart; and generate a device for implementing the functional actions specified in each block or the combination of blocks in the block diagram.

[0109] It should be understood that the processor in the computer can be understood as one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components implemented to execute the aforementioned computer instructions.

[0110] Computer instructions can be executed completely on the user's local computer, partially on the user's local computer, as a separate software package, partially on the user's local computer and partially on a remote computer, or completely on a remote computer or server. It should also be noted that in some alternative embodiments, the functions noted in each step in the flow chart or each block in the block diagram may not occur in the order noted in the figure. For example, depending on the functions involved, two steps or two blocks shown in succession may actually be executed roughly simultaneously, or these blocks may sometimes be executed in reverse order.

[0111] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. An agent-based data consistency audit system, characterized in that: include: An agent building module, wherein the agent building module builds a collection agent for a data source, and the collection agent extracts data to be checked from the data source; A feature extraction module, the feature extraction module is connected to the agent construction module, and the feature extraction module extracts data features from the data to be checked; A consistency calculation module, wherein the consistency calculation module is connected to the feature extraction module, and the consistency calculation module calculates data consistency according to the data features.

2. The data consistency audit system according to claim 1, characterized in that: The agent building module includes: A data source sniffing module, wherein the data source sniffing module collects data interface information from the data source; A connection establishment module, the connection establishment module is connected to the data source sniffing module, and the connection establishment module determines a connection protocol according to the interface information; A monitoring module, wherein the monitoring module is connected to the connection establishment module, and the monitoring module obtains data changes in the data source, and obtains the data to be checked via the connection protocol when data changes are triggered.

3. The data consistency audit system according to claim 1, characterized in that: The feature extraction module comprises: A data type determination module, wherein the data type determination module determines the data type of the data to be checked; A numerical data processing module, the numerical data processing module is connected to the data type determination module, and the numerical data processing module extracts the data features from the numerical data; A text data processing module, the text data processing module is connected to the data type determination module, and the text data processing module extracts the data features from the text data; A time data processing module, the time data processing module is connected to the data type judgment module, and the time data processing module extracts the data features from the time data.

4. The data consistency audit system according to claim 3, characterized in that: The numerical data processing module comprises: A set mean calculation module, wherein the set mean calculation module calculates the set mean of the numerical data; A data feature calculation module, wherein the data feature calculation module is connected to the set mean calculation module, and the data feature calculation module calculates the standard deviation, skewness and kurtosis according to the set mean as the data feature.

5. The data consistency audit system according to claim 3, characterized in that: The text data processing module comprises: A word segmentation module, wherein the word segmentation module performs word segmentation on the text data to obtain a word segmentation result; A word frequency statistics module, the word frequency statistics module is connected to the word segmentation module, and the word frequency statistics module constructs a word frequency vector according to the word segmentation result; A word frequency analysis module, wherein the word frequency analysis module is connected to the word frequency statistics module, and the word frequency analysis module obtains data features according to the word frequency vector analysis.

6. The data consistency audit system according to claim 1, characterized in that: The consistency calculation module includes: A consistency matrix construction module, wherein the consistency matrix construction module calculates the consistency of the classification data respectively according to the data features; An overall consistency calculation module, wherein the overall consistency calculation module is connected to the consistency matrix construction module, and the overall consistency calculation module calculates the data consistency according to the classification data consistency.

7. The data consistency audit system according to claim 6, characterized in that: The consistency matrix building module includes: A numerical consistency matrix construction module, wherein the numerical consistency matrix construction module constructs a data consistency matrix for the data features of numerical types and calculates the classification data consistency; A text consistency matrix construction module, wherein the text consistency matrix construction module constructs a data consistency matrix for the data features of the text type and calculates the classification data consistency; A time consistency matrix construction module is used to construct a data consistency matrix for the data features of the time type and calculate the classification data consistency.

8. The data consistency audit system according to claim 6, characterized in that: The overall consistency calculation module performs weighted calculation on the classification data consistency to obtain the data consistency.

9. The data consistency audit system according to claim 1, characterized in that: Also includes: An execution module, wherein the execution module is connected to the consistency calculation module, and the execution module calls a corresponding processing strategy according to the data consistency and the data type of the data source.

Images