Data standard penetration method and device, electronic equipment, medium and program product

Abstract

This application provides a data standardization method, apparatus, electronic device, medium, and program product based on multi-agent collaboration, involving the application of large models in fintech scenarios, and applicable to the fields of big data technology and artificial intelligence technology. The method includes: using a rule understanding agent to obtain a data standard document; parsing the data standard document to generate a structured rule base; using a data mapping agent to obtain a target data source; performing rule matching on the target data source based on the structured rule base to obtain candidate data items; and using a cleaning execution agent to perform cleaning operations on the candidate data items to generate standardized data.

Description

Technical Field

[0001] This application relates to the fields of big data technology and artificial intelligence technology, to the application of large models in financial technology scenarios, and more specifically to a data standard implementation method, device, equipment, medium and program product based on multi-agent collaboration. Background Technology

[0002] In various aspects of financial institutions' operations management, risk control modeling, and regulatory reporting, to ensure data consistency, integrity, and compliance, the banking industry generally conducts data standardization work based on multi-agent collaboration. This involves aligning heterogeneous data collected from multiple sources in business systems to a unified data standard system. However, existing technologies still face many limitations in data standardization practices.

[0003] Currently, banking institutions largely rely on manual methods or basic tools for standardization processes, primarily maintaining data standards in documented form and manually comparing business data with standard rules for mapping and correction. While some technical solutions have introduced rule engines or database scripts to achieve partial automation, they still suffer from the following shortcomings: First, the digitization level of data standards is low; standard documents are mostly described in natural language, lacking structured representation and unable to be directly converted into executable rules. Second, existing solutions have limited processing capabilities for heterogeneous data sources, mainly targeting structured data and struggling to adapt to the large amount of semi-structured and unstructured data present in banking operations. Third, the standardization process lacks sufficient intelligence, relying on manual operation and review, resulting in low processing efficiency and untimely response. Summary of the Invention

[0004] In view of the above problems, this application provides a data standardization method, apparatus, device, medium and program product based on multi-agent collaboration.

[0005] According to a first aspect of this application, a data standardization method based on multi-agent collaboration is provided. The method includes: using a rule understanding agent to obtain a data standard document, parsing the data standard document to generate a structured rule base; using a data mapping agent to obtain the structured rule base and a target data source, performing rule matching on the target data source based on the structured rule base to obtain candidate data items; and using a cleaning execution agent to perform cleaning operations on the candidate data items to generate standardized data.

[0006] According to an embodiment of this application, the method further includes: during the processing of the rule understanding agent, the data mapping agent, and the cleaning execution agent, at least recording the acquisition source of the data standard document and the target data source, the correspondence between the candidate data items and the standardized data, and the processing steps and execution subject information of any agent, to obtain audit records.

[0007] According to an embodiment of this application, the step of using a rule-understanding agent to acquire data standard documents, parsing the data standard documents, and generating a structured rule base includes: acquiring data standard documents from multiple business systems; performing semantic analysis on the data standard documents to obtain rule fragments, wherein the rule fragments include at least one of data standard field definitions, data standard value constraints, and data standard logical judgment conditions; constructing rule vector representations based on the rule fragments; performing cluster analysis on the rule vector representations to obtain similar rule groups and corresponding generalization labels; and constructing the structured rule base based on the similar rule groups and corresponding generalization labels.

[0008] According to an embodiment of this application, the step of using a data mapping agent to obtain a target data source and performing rule matching on the target data source based on the structured rule base to obtain candidate data items includes: parsing the data in the target data source to extract candidate fields and candidate field values; and matching the candidate fields and candidate field values ​​with the rule fragments to determine the candidate data items based on the matching results.

[0009] According to an embodiment of this application, the step of using a cleaning execution agent to perform cleaning operations on the candidate data items to generate standardized data includes: determining the processing conditions of the candidate data items; in response to the candidate data items meeting a first processing condition, performing a standardized transformation on the candidate data items based on the structured rule base to generate the standardized data; or in response to the candidate data items meeting a second processing condition, marking the candidate data items as manually intervened items and submitting them to a manual review process to generate the standardized data.

[0010] According to an embodiment of this application, the method further includes: using a feedback learning agent to collect feedback information from the cleaning execution agent performing cleaning operations; constructing sample pairs of abnormal data and processing strategies based on the feedback information; and training a rule optimization model in the feedback learning agent based on the samples, and updating the structured rule base using the trained rule optimization model.

[0011] According to an embodiment of this application, the method further includes: in response to the target data source being a streaming data source, dividing the target data source into multiple data sub-tasks according to a preset data sharding rule; and using the data mapping agent and the cleaning execution agent to perform rule matching and cleaning operations on the multiple data sub-tasks in an asynchronous collaborative manner.

[0012] According to an embodiment of this application, the method further includes: configuring role identifiers for the multiple agents respectively, establishing an access authorization policy based on the role identifiers; and, in response to any agent's request to access the structured rule base, the target data source, the candidate data items, and / or the standardized data, performing permission verification according to the access authorization policy.

[0013] A second aspect of this application provides a data standardization device based on multi-agent collaboration. The device includes: a rule understanding module, configured to: acquire a data standard document using a rule understanding agent, parse the data standard document, and generate a structured rule base; a data mapping module, configured to: acquire the structured rule base and a target data source using a data mapping agent, perform rule matching on the target data source based on the structured rule base, and obtain candidate data items; and a cleaning execution module, configured to: perform cleaning operations on the candidate data items using a cleaning execution agent, and generate standardized data.

[0014] According to embodiments of this application, the rule understanding module can also be used to acquire data standard documents from multiple business systems; perform semantic analysis on the data standard documents to obtain rule fragments, wherein the rule fragments include at least one of data standard field definitions, data standard value constraints, and data standard logical judgment conditions; construct rule vector representations based on the rule fragments; perform cluster analysis on the rule vector representations to obtain similar rule groups and corresponding generalization labels; and construct the structured rule base based on the similar rule groups and corresponding generalization labels.

[0015] According to an embodiment of this application, the data mapping module can also be used to parse and process the data in the target data source, extract candidate fields and candidate field values; and match the candidate fields and candidate field values ​​with the rule fragment, and determine the candidate data item based on the matching result.

[0016] According to embodiments of this application, the cleaning execution module can also be used to determine the processing conditions of the candidate data items; in response to the candidate data items meeting the first processing condition, to perform standardized transformation on the candidate data items based on the structured rule base to generate the standardized data; or in response to the candidate data items meeting the second processing condition, to mark the candidate data items as manually intervened items and submit them to the manual review process to generate the standardized data.

[0017] According to embodiments of this application, the data standardization device based on multi-agent collaboration may further include an audit module. The audit module can be used to record, at least during the processing of the rule understanding agent, the data mapping agent, and the cleaning execution agent, the source of acquisition of the data standard document and the target data source, the correspondence between the candidate data items and the standardized data, and the processing steps and execution entity information of any agent, thereby obtaining audit records.

[0018] According to embodiments of this application, the data standardization device based on multi-agent collaboration may further include a feedback learning module. The feedback learning module can be used to collect feedback information from the cleaning execution agent during cleaning operations using a feedback learning agent; construct sample pairs of abnormal data and processing strategies based on the feedback information; train a rule optimization model in the feedback learning agent based on the samples; and update the structured rule base using the trained rule optimization model.

[0019] According to embodiments of this application, the data standardization device based on multi-agent collaboration may further include a stream processing module. The stream processing module can be used to, in response to the target data source being a streaming data source, to divide the target data source into multiple data sub-tasks according to preset data sharding rules; and to utilize the data mapping agent and the cleaning execution agent to perform rule matching and cleaning operations on the multiple data sub-tasks in an asynchronous collaborative manner.

[0020] According to embodiments of this application, the data standardization device based on multi-agent collaboration may further include a permission verification module. The permission verification module can be used to configure role identifiers for each of the multiple agents, establish access authorization policies based on the role identifiers, and, in response to any agent's request to access the structured rule base, the target data source, the candidate data items, and / or the standardized data, perform permission verification according to the access authorization policy.

[0021] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0022] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0023] According to embodiments of this application, by independently assigning rule parsing, data matching, and cleaning processing to rule understanding agents, data mapping agents, and cleaning execution agents respectively, and employing a collaborative mechanism to complete the standardization process, intelligent and automated execution of each processing task is achieved. This reduces the cost of manual intervention, improves the processing throughput and task response speed of the standardization system, and enhances the system's adaptability in high-concurrency and real-time scenarios. Simultaneously, by automatically identifying candidate data items and performing standardized cleaning based on the rule base, the tedious process of manual field-by-field review and confirmation is avoided, lowering the barrier to entry and reducing the risk of errors. This provides data governance personnel with a highly transparent and low-interference operational experience. Attached Figure Description

[0024] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0025] Figure 1 The illustration shows an application scenario of a data standardization method, apparatus, device, medium, and program product based on multi-agent collaboration according to embodiments of this application.

[0026] Figure 2 A flowchart illustrating a data standardization method based on multi-agent collaboration according to an embodiment of this application is shown schematically.

[0027] Figure 3 The flowchart illustrates a method for parsing using a rule-understanding agent according to some exemplary embodiments of this application.

[0028] Figure 4 The flowchart illustrates a method for rule matching using a data mapping agent according to some exemplary embodiments of this application.

[0029] Figure 5 This schematically illustrates a structural block diagram of a data standardization device based on multi-agent collaboration according to an embodiment of this application; and

[0030] Figure 6 A block diagram of an electronic device suitable for implementing a data standardization method based on multi-agent collaboration, according to an embodiment of this application, is illustrated schematically. Detailed Implementation

[0031] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0034] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0035] First, the technical terms used in this article are explained and clarified as follows.

[0036] Data standardization refers to the process of standardizing the format, aligning fields, validating the validity of values, and converting semantics of data from different systems and sources according to unified data standards.

[0037] Standardized data refers to output data that meets the requirements of data standards and specifications after being processed through the standardization process. It has completed processing operations such as field format conversion, missing value filling, and unit unification, and has a compliant and usable structured form.

[0038] An agent is a functional entity in a system that undertakes a specific task and possesses the ability to execute autonomously, respond to tasks, and communicate collaboratively. Each agent can be viewed as a subsystem with specific input / output interfaces and decision-making logic.

[0039] Streaming data refers to data generated and transmitted in a real-time, continuous, and high-frequency manner, and is commonly found in scenarios such as financial transaction records, user operation logs, and sensor data.

[0040] In key business processes across financial institutions, including operations management, risk control modeling, internal auditing, regulatory reporting, and data asset management, the banking industry widely implements data standardization practices to ensure the consistency, integrity, and compliance of cross-system data. Data standardization refers to standardizing, semantically aligning, and verifying the quality of multi-source, heterogeneous data scattered across different business systems according to established data standards, thereby forming a unified data representation and shared semantic foundation. This work not only supports the improvement of internal data governance capabilities but also lays the foundation for meeting external regulatory requirements, possessing significant institutional and strategic importance.

[0041] However, limited by current technological means and the maturity of data governance systems, existing data standardization practices still face multiple challenges. Specifically:

[0042] Firstly, regarding the level of digitization of data standards, financial institutions currently generally adopt a document-based standard maintenance approach, which involves storing data specifications, field definitions, and value rules in text files. These standard documents, described in natural language, lack structured semantics and are difficult to directly translate into machine-readable and executable rule formats. This results in the standardization process heavily relying on human understanding and interpretation, leading to low efficiency and a high error rate.

[0043] Secondly, regarding heterogeneous data processing capabilities, most current standardization tools and methods focus on structured data processing in relational databases, while their support for semi-structured data (such as JavaScript Object Notation (JSON) formatted transaction messages and configuration files) and unstructured data (such as log records, graphic reports, image-to-text recognition, and knowledge graph node relationships) widely existing in actual business systems is weak. This makes it difficult to incorporate a large amount of critical data into the standardization system, limiting the breadth and depth of standard governance.

[0044] Secondly, regarding the level of process intelligence and automation, although some systems have introduced rule engines or database scripts to attempt partial automation, they still struggle to escape the dominance of human intervention. The standardization process typically relies on data governance personnel or business experts to manually complete rule matching, field mapping, and anomaly identification, resulting in overall inefficiency and difficulty in handling massive amounts of data and frequently updated scenarios. Furthermore, the standardization process lacks a full-chain recording and traceability mechanism, failing to meet the financial industry's requirements for data transparency, process auditability, and regulatory compliance.

[0045] Based on this, embodiments of this application provide a data standardization method based on multi-agent collaboration, including: using a rule understanding agent to obtain a data standard document, parsing the data standard document to generate a structured rule base; using a data mapping agent to obtain a target data source, performing rule matching on the target data source based on the structured rule base to obtain candidate data items; and using a cleaning execution agent to perform cleaning operations on the candidate data items to generate standardized data. According to embodiments of this application, by independently assigning rule parsing, data matching, and cleaning processing to the rule understanding agent, data mapping agent, and cleaning execution agent respectively, and employing a collaborative mechanism to complete the standardization process, intelligent and automated execution of each processing task is achieved, reducing the cost of manual intervention, improving the processing throughput and task response speed of the standardization system, and enhancing the system's adaptability in high-concurrency, real-time scenarios. Simultaneously, by automatically identifying candidate data items and performing standardized cleaning based on the rule base, the tedious process of manual field-by-field review and confirmation is avoided, lowering the barrier to entry and reducing the risk of errors, providing data governance personnel with a highly transparent and low-interference operating experience.

[0046] It should be noted that the data standardization method, apparatus, device, medium, and program products based on multi-agent collaboration identified in this application relate to the application of large models in fintech scenarios. They can be used in the fields of big data technology and artificial intelligence technology, as well as in the fintech field and various other fields besides big data technology, artificial intelligence technology, and fintech. The application areas of the data standardization method, apparatus, device, medium, and program products based on multi-agent collaboration provided in the embodiments of this application are not limited.

[0047] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0048] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0049] Figure 1 The illustration shows an application scenario of a data standardization method, apparatus, device, medium, and program product based on multi-agent collaboration according to embodiments of this application.

[0050] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0051] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0052] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0053] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0054] It should be noted that the data standardization method based on multi-agent collaboration provided in this application embodiment can generally be executed by server 105. Correspondingly, the data standardization device based on multi-agent collaboration provided in this application embodiment can generally be located in server 105. The data standardization method based on multi-agent collaboration provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data standardization device based on multi-agent collaboration provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0055] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0056] The following will be based on Figure 1 The described scene, through Figures 2-4 The data standardization method based on multi-agent collaboration based on the disclosed embodiments is described in detail.

[0057] Figure 2 A flowchart illustrating a data standardization method based on multi-agent collaboration according to an embodiment of this application is shown.

[0058] like Figure 2 As shown, the data standardization method 200 based on multi-agent collaboration in this embodiment includes operations S210 to S230.

[0059] In operation S210, the rule-understanding agent obtains a data standard document, parses the data standard document, and generates a structured rule base.

[0060] In embodiments of this application, the data standard document may include a data standard specification document used within the financial industry to describe the field attributes, format specifications, value ranges, and logical relationships of business data. The data standard document may be provided in natural language or tabular form, and is characterized by strong semantic diversity, loose structure, and inability to be directly executed by machines.

[0061] In the embodiments of this application, the rule understanding agent can obtain data standard documents by accessing a document management platform, an interface configuration system, or a preset data standard directory. After obtaining the data standard documents, the rule understanding agent can process the document content through an embedded parsing engine to extract structural descriptive information related to the data fields. For example, for tabular standard documents, the system can identify explicit field items such as "field name," "data type," "required field," and "value description"; for natural language documents, the system can extract text fragments with rule meaning through semantic segmentation, keyword recognition, template matching, and other methods.

[0062] For example, a rule-understanding agent can use a pre-defined set of parsing rules or a template library to extract unified field descriptions from documents of different formats. For instance, for expressions in a business system description such as "account opening date must be a past time", "mobile phone number should be 11 digits", and "transaction status should be 'successful', 'failed', or 'processing'", the rule-understanding agent can map them into structured rules such as time constraints, format constraints, and enumeration value restrictions, respectively.

[0063] After parsing a document, the rule-understanding agent can organize the extracted field information, relationships between fields, and constraints into structured rule entries, forming a structured rule library. The structured rule library can be stored in JSON format, tables, databases, etc. Each rule includes information such as field identifiers, scope of application, rule type (e.g., format, value range, logical conditions), and execution parameters.

[0064] To enhance the versatility of the rule base, the system can also identify the context of rules during document parsing, such as indicating which business system, data table, field group, or processing scenario the rule applies to, supporting on-demand invocation in subsequent standardization processes. After generating a structured rule base, the rule understanding agent can push the structured rule base to the data mapping agent for data field comparison and anomaly identification.

[0065] In operation S220, the data mapping agent is used to obtain the structured rule base and the target data source, and the target data source is matched with rules based on the structured rule base to obtain candidate data items.

[0066] In the embodiments of this application, the application process of standardization rules can be executed by a data mapping agent, specifically including obtaining a structured rule base and a target data source, and performing field-by-field matching, verification and filtering of the target data source based on the rule content, so as to identify candidate data items that have differences or do not meet the standard requirements.

[0067] The target data source can come from database tables, API return data, report output, or historical log files from multiple business systems within the bank. Data types may include structured data (such as relational database fields), semi-structured data (such as JSON format APIs), or even field records extracted from documents or tables. The data mapping agent supports unified parsing of these different data sources. For example, through a pre-defined data decoder or field location strategy, it converts the raw data into a set of field-value pairs that can be used for rule comparison.

[0068] After receiving a structured rule base, the data mapping agent can compare the corresponding fields in the target data source one by one, according to the field names, data types, value ranges, and format constraints defined in the rules. For example, if the rule base defines "the ID number field should be 18 digits", the system will search for matching field names in the target data source and check whether their field values ​​meet the length, character type, and validation rule requirements. As another example, if a rule defines that a field should be limited to "male" or "female", the system will enumerate and match the actual field values ​​to identify variations of "male", "female", "M / F", or invalid values.

[0069] In some embodiments, the data mapping agent can not only perform static field mapping, but also combine the context conditions in the rules to realize multi-field association verification. It can retrieve the values ​​of multiple fields at the same time and perform combined verification when the conditions are met, so as to not only detect single-field anomalies, but also identify potential rule conflicts between fields.

[0070] In multi-system parallel processing scenarios, the data mapping agent can also execute a unified mapping strategy on data sources from different business systems and handle matching failures caused by naming inconsistencies through field mapping tables or alias rules. For example, different systems may name the "transaction time" field as "aaa_time" or "bbb_time" respectively. The system can use field labels, business context, or pre-trained models to perform field semantic normalization, thereby improving the accuracy of field recognition and mapping.

[0071] Based on this, the data mapping agent can identify data items that do not meet the rule constraints and mark them as candidate data items. Candidate data items can include various types such as missing fields, format errors, inconsistent units, logical conflicts, and out-of-bounds values. The system can generate a structured anomaly report for each candidate item, including the field name, original value, the violated rule number or description, anomaly type, and location information, for use in subsequent cleaning phases.

[0072] In operation S230, the cleaning execution agent is used to perform a cleaning operation on the candidate data items to generate standardized data.

[0073] In the embodiments of this application, the cleaning execution agent can automatically determine the anomaly type and repair method of each candidate data item based on information in the structured rule base.

[0074] For example, if a field should be a date in the format "YYYY-MM-DD", but candidate data items contain formats such as "20250709" or "09 / 07 / 2025", the cleaning agent can directly perform date format conversion operations according to standard rules.

[0075] For example, if the rules set default values ​​or derivation rules, such as "the currency field is empty by default to CNY" or "if the account opening branch is missing, the common values ​​of other records under the same account number will be inherited", the cleaning execution agent can automatically fill in the missing values.

[0076] For example, for candidates with inconsistent units, the cleaning execution agent can perform unit conversion according to field rules, such as converting the "amount" field from "ten thousand yuan" to "yuan", or converting the "interest rate" field from percentage to decimal expression, to ensure data alignment under a unified standard.

[0077] For example, if the anomaly of the candidate data item comes from a data type error, such as a boolean field showing "yes / no" instead of "true / false", the cleaning agent can execute a value mapping replacement strategy.

[0078] For example, when the enumerated value is not in the list of valid values, the cleaning agent can also provide correction capabilities based on fuzzy matching or approximate standards, such as unifying "processing" and "processing" into "processing".

[0079] In real-world business environments, some candidate data items may not be automatically processed by rules, such as those with semantically ambiguous field values, contextual dependencies, or exceptions not covered by existing rules. In such cases, the data cleaning agent can mark these candidate items as requiring manual intervention, generate a work order, and submit it to the manual review workflow. The system includes the field values, the matched rules, an anomaly description, and recommended repair strategies in the work order for manual judgment and supplementation, ensuring that the data repair process in complex scenarios has business controllability and a closed-loop processing mechanism.

[0080] During the data cleaning process, the cleaning agent generates a corresponding processing record for each data item, recording information such as field values ​​before and after processing, applied rule numbers, operation types, and execution times. These processing records can be used for subsequent data quality assessments, process reviews, and problem tracing.

[0081] Furthermore, the cleaning execution agent can also support flexible switching between batch processing mode and item-by-item processing mode, and can automatically adjust the scheduling strategy according to the size of the data source, processing complexity, and system performance. For scenarios with a large number of candidate data items, the cleaning execution agent can divide the cleaning task into multiple processing sub-blocks and execute them concurrently to improve processing efficiency; while for business-critical fields with high accuracy requirements, an item-by-item confirmation strategy can be configured to ensure the accuracy and compliance of data processing.

[0082] In the embodiments of this application, in order to ensure the traceability, compliance and auditability of the entire data standardization process, an audit recording mechanism can be introduced in the processing of the rule understanding agent, the data mapping agent, and the cleaning execution agent to uniformly record and structurally store the inputs, outputs and processing behaviors of key processing links.

[0083] Specifically, during the processing of the rule understanding agent, the source information of the acquired data standard documents can be recorded, including the document file name, version number, providing system, upload time, and parsing timestamp; at the same time, the structured rule entries generated by parsing, the corresponding original text fragments, and the mapping relationship can be recorded, which is convenient for locating the basis for rule changes in the standard evolution process.

[0084] During the execution of the data mapping agent, the source information of the target data source can be recorded, including metadata such as data file name, system module, collection time, and interface call identifier. For each candidate data item, its field position in the original data, original value, matching rule identifier, and anomaly type classification are recorded. At the same time, the system can establish a mapping relationship between candidate data items and subsequent cleaning results, forming a complete process link of "abnormal fields corresponding to standardized output".

[0085] During the cleaning process, the system can record the rule number, processing strategy type (such as format conversion, missing data filling, unit conversion, manual intervention, etc.), changes in field values ​​before and after processing, execution time, and cleaning result status (such as successful processing, transferred to manual review, etc.) for each cleaning operation. If manual intervention is involved, the system can further record the processor's identifier, operational comments, and final confirmation result to ensure the verifiability of the cleaning decision path.

[0086] In addition, it can uniformly record information such as task number, task execution order, participating module name, processing start and end time, and execution entity information (such as service instance identifier, user account, and processing role) for each intelligent agent's processing process, forming a task-level audit trail view. All audit records will be archived in chronological order and stored in the audit log database or data lake audit area, allowing authorized supervisory, auditing, or quality management personnel to query and analyze them as needed.

[0087] According to embodiments of this application, audit logs not only provide transparency in the data processing process, but also have the ability to trace any abnormal results back to their source data, rules, and processing procedures, thereby enhancing the credibility and regulatory compliance of the standardization process and meeting the compliance requirements of the financial industry in data governance and risk control.

[0088] The following will describe in detail the data standardization method based on multi-agent collaboration of this application through preferred embodiments.

[0089] Figure 3 The flowchart illustrates a method for parsing using a rule-understanding agent according to some exemplary embodiments of this application.

[0090] like Figure 3 As shown, the method for parsing using a rule-understanding agent includes operations S310 to S340.

[0091] When operating S310, data standard documents from multiple business systems are obtained.

[0092] In the embodiments of this application, multiple business systems may include multiple data source systems within a financial institution, such as a personal banking system, a corporate banking system, a credit approval system, a payment and settlement system, a risk control system, a customer management system, and a reporting and regulatory system. During the process of acquiring standard data documents from multiple business systems, the documents can be labeled with their attribution to record metadata such as the business system identifier, version information, effective date, and responsible person for each document, enabling subsequent rule differentiation processing and version management. For example, documents from the "credit approval system" may focus on fields such as loan agreements, credit lines, and credit limits, while documents from the "payment and settlement system" may focus more on fields such as transaction time, settlement channel, and channel identifier. Rules can be categorized and organized according to business lines.

[0093] In operation S320, semantic analysis is performed on the data standard document to obtain rule fragments, which include at least one of data standard field definitions, data standard value constraints, and data standard logical judgment conditions.

[0094] In the embodiments of this application, the semantic analysis process can be based on natural language processing technology to identify descriptive sentences in the document related to field definitions, field value restrictions, and logical relationships between fields. For example, for statements such as "customer level should be A, B, C, or D" and "account opening date must not be later than the current date," the system can extract value constraints and time logic judgment rules respectively. The extracted rule fragments can carry metadata such as the original statement, associated fields, and rule type.

[0095] In the embodiments of this application, the data standard field definition can be used to describe basic attributes such as field name, type, unit, length, and whether it is required; the data standard value constraint can be used to limit the legal value range or enumeration set of the field; the data standard logical judgment condition can involve the conditional dependency or mutual exclusion relationship between multiple fields. The rule understanding agent can provide an executable rule basis for subsequent data matching, anomaly identification, and consistency verification by uniformly extracting and structurally representing the above different types of rules.

[0096] In operation S330, a rule vector representation is constructed based on the rule fragment, and cluster analysis is performed on the rule vector representation to obtain similar rule groups and corresponding generalized labels.

[0097] In the embodiments of this application, the rule vector representation can be implemented by an embedding model, a sentence vector generator, or a dedicated rule encoding network, with the aim of uniformly mapping rule fragments with diverse semantic structures to a low-dimensional vector space for similarity measurement.

[0098] Due to differences in historical development paths, data model design philosophies, usage scenarios, and technology stacks across different business systems, independent data standard documents have emerged, exhibiting variations and even conflicts in field definitions, naming conventions, formatting requirements, and logical rules. Therefore, after vectorizing rule fragments, the rule understanding agent can perform cluster analysis on the rule vector set to identify semantically similar or semantically consistent rule expressions from different business systems. The clustering results can identify common patterns among rules, such as classifying "document type is ID card" and "document type must be 'ID card' or 'passport'" as belonging to the same semantic category.

[0099] In operation S340, the structured rule base is constructed based on the similar rule groups and the corresponding generalized tags.

[0100] Generalized labels can be understood as abstract representations of a class of rules at the semantic level, such as "fields are restricted to fixed enumeration values" or "fields must be earlier than a certain point in time." Rule-understanding agents can use generalized labels to form standard rule templates, and then fill in specific instantiated rules into the template structure to complete a unified structured representation.

[0101] Figure 4 The flowchart illustrates a method for rule matching using a data mapping agent according to some exemplary embodiments of this application.

[0102] like Figure 4 As shown, the method for rule matching using a data mapping agent includes operations S410 to S420.

[0103] In operation S410, the data in the target data source is parsed and processed to extract candidate fields and candidate field values.

[0104] During the parsing process, the data mapping agent can perform synonym normalization, case standardization, and prefix / suffix removal on field names to improve the accuracy of field identification. For example, for fields with different names in different systems, the data mapping agent can identify them as candidate instances of the same standard field. Regarding field values, the parsing process retains the original numerical values ​​while also extracting format information, unit information, or contextual modifiers for subsequent comparison.

[0105] In operation S420, the candidate fields and their values ​​are matched with the rule fragments, and the candidate data items are determined based on the matching results.

[0106] In the embodiments of this application, the matching process may include multiple levels such as field name matching, data type matching, format matching, value matching, and logical constraint judgment. For cases with logical rule constraints, the data mapping agent can also perform cross-field matching and joint verification.

[0107] For example, when the rule is defined as "if the account status is frozen, the transaction status must not be successful", the data mapping agent will perform a combined judgment when multiple field values ​​meet the triggering conditions. Any field instance that violates the field definition, value range requirements, or logical constraints can be marked as a candidate data item and form a structured anomaly identifier, i.e., a candidate data item.

[0108] In the embodiments of this application, after receiving candidate data items, the cleaning execution agent can first perform a processing condition judgment on each candidate data item to determine whether the standardization operation can be automatically completed based on the structured rule base, or whether manual intervention is required. This judgment process can be executed according to preset rule adaptation logic, data integrity characteristics, and processing feasibility strategies, ensuring that the cleaning execution agent can achieve a reasonable division of labor between efficient automatic cleaning and necessary manual correction while ensuring the accuracy of standardization.

[0109] For candidate data items that meet the first processing condition, the cleaning agent can directly perform standardized transformation operations based on the definition of the corresponding rule entry in the structured rule base. The first processing condition may refer to the existence of explicit and mappable repair rules for the candidate data item, or data anomalies such as format, units, and symbols that can be corrected through predefined logic.

[0110] For candidate data items that do not meet the first processing condition but meet the second processing condition, the data cleaning agent can mark them as items requiring manual intervention and submit the relevant information to the manual review process. The second processing condition may be that the anomaly of the candidate data item exceeds the current expressive capacity of the rule base, or that there are complex issues with semantic ambiguity, contextual dependence, or strong business judgment logic, making it impossible for the system to automatically determine the repair strategy. For example, the value of "customer status is unknown" cannot be clearly mapped to a standard category, or "credit rating is empty but the customer is a newly opened enterprise" has business exception logic. In such cases, the system will generate a pending work order, which includes the original field value, triggering rule, anomaly type description, and possible repair suggestions, for manual confirmation and processing by data governance personnel or business experts.

[0111] According to embodiments of this application, the cleaning execution agent can implement a dual-track processing strategy based on intelligent judgment, which ensures processing efficiency while retaining the ability to intervene manually in complex problems, ensuring that the cleaning process is both automated and does not lose business accuracy and audit transparency.

[0112] In some embodiments, a feedback learning agent can be introduced to continuously learn the cleaning effect during the data standardization process and dynamically optimize the rule system. The feedback learning agent can collect feedback information from the cleaning execution agent during actual cleaning operations, construct correlation samples between abnormal data and processing strategies, and train the rule optimization model within the feedback learning agent based on this, thereby continuously improving the adaptability and coverage of the structured rule base.

[0113] Specifically, the feedback learning agent can collect feedback information in real time during the data cleaning execution phase, including the anomaly type of candidate data items, the cleaning processing path, whether the repair was successful, the rule number used, the results of manual processing, the operation time, and the final output. For items requiring manual intervention, it can also collect the repair choices, notes, and confirmation opinions made by data governance personnel during the review process. This information, after being aggregated, can form a pair of "anomaly data samples and processing strategies" for training, reflecting the system's experiential paths in handling similar problems on historical data.

[0114] After constructing the training samples, the feedback learning agent can invoke the built-in rule optimization model to perform supervised learning or policy optimization. The rule optimization model can employ algorithmic frameworks such as classifiers, matchers, sequence modelers, and graph neural networks to learn the mapping relationship between different anomaly types and their optimal handling solutions. After training, the rule optimization model can provide repair suggestions for anomalies that are not yet covered or are poorly identified, or it can identify problems such as insufficient precision or coverage in the original rules, thereby forming supplementary entries, optimization suggestions, or replacement schemes for structured rules.

[0115] Furthermore, the feedback learning agent can use the output of the trained model to update the structured rule base. Update methods can include adding new rule fragments, modifying rule condition parameters, expanding the scope of rule application, or recommending rule merging strategies. The update process can be configured to be automatic, take effect after manual review and confirmation, or prompt the rule administrator for intervention and confirmation in a suggestion mode, ensuring the controllability and compliance of the rule evolution process.

[0116] By introducing a feedback learning agent and constructing an optimization closed loop based on cleaning practice data, the embodiments of this application realize the self-learning and self-updating capabilities of the rule base, reduce the reliance on manual rule maintenance, and improve the overall intelligence level and long-term operational stability of the system.

[0117] In the embodiments of this application, for streaming data processing scenarios, a parallel standardization mechanism based on task splitting and asynchronous collaboration can also be executed to improve the system's processing throughput and response performance under high-frequency data input.

[0118] Specifically, when a streaming data source is detected as the target data source, the streaming processing mode can be automatically initiated, and the input data can be segmented into tasks at the operation entry point according to preset data segmentation rules. These data segmentation rules can be flexibly configured based on strategies such as time windows (e.g., batching by seconds or minutes), data fields (e.g., segmenting by customer number, account number, or organization number), and the number of records (e.g., 1000 records per block). Each segmented data block constitutes an independent data subtask, possessing complete field information and serving as the smallest unit for standardization processing.

[0119] After task segmentation, data subtasks can be distributed to a data mapping agent and a cleaning execution agent for parallel processing. The data mapping agent first performs rule matching operations on the fields in each data subtask to identify candidate data items; subsequently, the cleaning execution agent performs standardization processing or manual labeling on the identified candidate data items. Throughout the process, the processing flow of each subtask can be carried out asynchronously among multiple agents, avoiding blocking waits and serial bottlenecks, and improving processing efficiency.

[0120] To ensure the accuracy and consistency of asynchronous execution, task scheduling and state tracking mechanisms can be introduced between agents. Each data subtask carries a unique task identifier and its source information when entering the processing flow. The system uses task queues, message buses, or distributed task coordination components to achieve orderly task scheduling and processing state management. After processing, the system can aggregate the results of multiple subtasks as needed to generate standardized outputs or exception reports.

[0121] Through the embodiments of this application, the standardization system can maintain high concurrency and high throughput processing capabilities when dealing with streaming data, avoiding processing bottlenecks caused by data accumulation or response delays. It is suitable for scenarios with high timeliness requirements, such as financial transactions, real-time risk control, and real-time reporting. At the same time, the parallel standardization mechanism based on task partitioning and asynchronous collaboration can fully accommodate the system's original rule matching and cleaning logic, avoiding functional coupling problems caused by changes in processing modes, and possessing good scalability and stability.

[0122] In the embodiments of this application, to ensure data access security and permission isolation during multi-agent collaboration in the data standardization system, a role-based access authorization mechanism can be introduced. This mechanism assigns independent role identifiers to different types of agents and establishes corresponding access authorization policies to achieve fine-grained access control of critical data resources, preventing unauthorized access and data leakage, and meeting the compliance requirements of the financial industry for data hierarchical management and operational auditing.

[0123] Specifically, during system initialization, each agent can be assigned a unique role identifier, which can be configured based on the agent's type, functional responsibilities, deployment environment, and scope of use. For example, a rule understanding agent can be assigned a rule reading role, primarily for accessing data standard documents and writing to the structured rule base; a data mapping agent can be assigned a data matching role, possessing permissions to read the rule base and target data sources; and a cleaning execution agent can be assigned a data cleaning role, capable of reading candidate data items and writing standardized data.

[0124] In actual operation, when any intelligent agent initiates an access request to sensitive resources, the system will match the access authorization policy based on its role identifier and perform permission verification. The access authorization policy can be preset as a whitelist, a role-based access control model, or a policy rule engine, and will be judged based on dimensions such as the access object, operation type (read / write / modify), and data range. If the access request meets the authorization conditions, the system will allow the request; if it does not meet the conditions, the system will deny the access and record the access failure event as part of the audit log.

[0125] To enhance the dynamism and configurability of access control, the system also supports hot updates, version management, and visual maintenance of access policies. Administrators can adjust the access scope of different agents based on business changes or security policies, such as restricting a certain type of agent to access only specific data source partitions during a specific time period, or adding additional audit tags to important fields.

[0126] According to the embodiments of this application, an access control framework centered on intelligent agent roles can be established to ensure that various intelligent agents can only access necessary data within their scope of responsibility, effectively preventing misreading, miswriting, or unauthorized data operation behaviors, and strengthening the data security, responsibility boundaries, and audit controllability of the system in a multi-module distributed collaborative environment.

[0127] Corresponding to the above-mentioned data standardization method based on multi-agent collaboration, embodiments of this application also provide a data standardization device based on multi-agent collaboration.

[0128] Figure 5 A schematic diagram of a data standardization device based on multi-agent collaboration according to an embodiment of this application is shown.

[0129] like Figure 5 As shown, the data standard implementation device 500 based on multi-agent collaboration in this embodiment includes a rule understanding module 510, a data mapping module 520, and a cleaning execution module 530.

[0130] The rule understanding module 510 can be used to acquire a data standard document using a rule understanding agent, parse the data standard document, and generate a structured rule base. In one embodiment, the rule understanding module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0131] The data mapping module 520 can be used to obtain the structured rule base and the target data source using a data mapping agent, and perform rule matching on the target data source based on the structured rule base to obtain candidate data items. In one embodiment, the data mapping module 520 can be used to execute the operation S220 described above, which will not be repeated here.

[0132] The cleaning execution module 530 can be used to perform cleaning operations on the candidate data items using a cleaning execution agent to generate standardized data. In one embodiment, the cleaning execution module 530 can be used to execute the operation S230 described above, which will not be repeated here.

[0133] According to an embodiment of this application, the rule understanding module 510 can also be used to obtain data standard documents from multiple business systems; perform semantic analysis on the data standard documents to obtain rule fragments, wherein the rule fragments include at least one of data standard field definitions, data standard value constraints, and data standard logical judgment conditions; construct rule vector representations based on the rule fragments; perform cluster analysis on the rule vector representations to obtain similar rule groups and corresponding generalization labels; and construct the structured rule base based on the similar rule groups and corresponding generalization labels.

[0134] According to an embodiment of this application, the data mapping module 520 can also be used to parse and process the data in the target data source, extract candidate fields and candidate field values; and match the candidate fields and candidate field values ​​with the rule fragments, and determine the candidate data items based on the matching results.

[0135] According to an embodiment of this application, the cleaning execution module 530 can also be used to determine the processing conditions of the candidate data items; in response to the candidate data items meeting the first processing conditions, to perform standardized transformation on the candidate data items based on the structured rule base to generate the standardized data; or in response to the candidate data items meeting the second processing conditions, to mark the candidate data items as manually intervened items and submit them to the manual review process to generate the standardized data.

[0136] According to embodiments of this application, the data standardization device 500 based on multi-agent collaboration may further include an audit module. The audit module can be used to record, at least during the processing of the rule understanding agent, the data mapping agent, and the cleaning execution agent, the source of acquisition of the data standard document and the target data source, the correspondence between the candidate data items and the standardized data, and the processing steps and execution entity information of any agent, thereby obtaining audit records.

[0137] According to an embodiment of this application, the data standardization device 500 based on multi-agent collaboration may further include a feedback learning module. The feedback learning module can be used to collect feedback information from the cleaning execution agent during cleaning operations using a feedback learning agent; construct sample pairs of abnormal data and processing strategies based on the feedback information; train a rule optimization model in the feedback learning agent based on the samples; and update the structured rule base using the trained rule optimization model.

[0138] According to an embodiment of this application, the data standardization device 500 based on multi-agent collaboration may further include a stream processing module. The stream processing module can be used to, in response to the target data source being a streaming data source, to divide the target data source into multiple data sub-tasks according to preset data sharding rules; and to utilize the data mapping agent and the cleaning execution agent to perform rule matching and cleaning operations on the multiple data sub-tasks in an asynchronous collaborative manner.

[0139] According to embodiments of this application, the data standardization device 500 based on multi-agent collaboration may further include a permission verification module. The permission verification module can be used to configure role identifiers for each of the multiple agents, establish access authorization policies based on the role identifiers, and, in response to any agent's request to access the structured rule base, the target data source, the candidate data items, and / or the standardized data, perform permission verification according to the access authorization policy.

[0140] According to embodiments of this application, any multiple modules among the rule understanding module 510, data mapping module 520, and cleaning execution module 530 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the rule understanding module 510, data mapping module 520, and cleaning execution module 530 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the rule understanding module 510, data mapping module 520, and cleaning execution module 530 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0141] Figure 6 A block diagram of an electronic device suitable for implementing a data standardization method based on multi-agent collaboration, according to an embodiment of this application, is illustrated schematically.

[0142] like Figure 6 As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0143] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0144] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0145] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0146] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0147] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the data standardization method based on multi-agent cooperation provided in the embodiments of this application.

[0148] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0149] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0150] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0151] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0153] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0154] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A data standardization method based on multi-agent collaboration, characterized in that, The method includes: A rule-understanding agent is used to obtain a data standard document, which is then parsed to generate a structured rule base. The data mapping agent is used to obtain the structured rule base and the target data source; rule matching is performed on the target data source based on the structured rule base to obtain candidate data items; and The candidate data items are cleaned using a cleaning agent to generate standardized data.

2. The method according to claim 1, characterized in that, The method further includes: During the processing of the rule understanding agent, the data mapping agent, and the cleaning execution agent, at least the acquisition source of the data standard document and the target data source, the correspondence between the candidate data items and the standardized data, and the processing steps and execution entity information of any agent are recorded to obtain audit records.

3. The method according to claim 1 or 2, characterized in that, The process of using a rule-understanding agent to acquire a data standard document, parsing the data standard document, and generating a structured rule base includes: Obtain data standard documents from multiple business systems; Perform semantic analysis on the data standard document to obtain rule fragments, wherein the rule fragments include at least one of data standard field definitions, data standard value constraints, and data standard logical judgment conditions; Based on the rule fragments, a rule vector representation is constructed; cluster analysis is performed on the rule vector representation to obtain similar rule groups and corresponding generalized labels; and The structured rule base is constructed based on the similarity rule groups and the corresponding generalized tags.

4. The method according to claim 3, characterized in that, The process of using a data mapping agent to obtain a target data source, and performing rule matching on the target data source based on the structured rule base to obtain candidate data items includes: The data in the target data source is parsed and processed to extract candidate fields and their values; and The candidate fields and their values ​​are matched with the rule fragments, and the candidate data items are determined based on the matching results.

5. The method according to claim 1, 2, or 4, characterized in that, The step of using a cleaning agent to clean the candidate data items and generate standardized data includes: The candidate data items are processed and conditionally judged; In response to the candidate data item meeting the first processing condition, the candidate data item is standardized and transformed based on the structured rule base to generate the standardized data; or in response to the candidate data item meeting the second processing condition, the candidate data item is marked as a manually intervened item and submitted to the manual review process to generate the standardized data.

6. The method according to claim 5, characterized in that, The method further includes: The system utilizes a feedback learning agent to collect feedback information from the cleaning execution agent during cleaning operations, and constructs sample pairs of abnormal data and processing strategies based on this feedback information; and The rule optimization model in the feedback learning agent is trained based on the samples, and the structured rule base is updated using the trained rule optimization model.

7. The method according to claim 1, characterized in that, The method further includes: In response to the target data source being a streaming data source, the target data source is divided into multiple data subtasks according to preset data sharding rules; and The data mapping agent and the cleaning execution agent are used to perform rule matching and cleaning operations on the multiple data subtasks in an asynchronous collaborative manner.

8. The method according to any one of claims 1, 2, 4, 6 and 7, characterized in that, The method further includes: Configure role identifiers for each of the multiple agents, and establish access authorization policies based on the role identifiers; and In response to any agent's request to access the structured rule base, the target data source, the candidate data items, and / or the standardized data, permission verification is performed according to the access authorization policy.

9. A data standard implementation device based on multi-agent collaboration, characterized in that, The device includes: The rule understanding module is used to: acquire data standard documents using a rule understanding agent, parse the data standard documents, and generate a structured rule base; The data mapping module is used to: acquire the structured rule base and the target data source using a data mapping agent; perform rule matching on the target data source based on the structured rule base to obtain candidate data items; and The cleaning execution module is used to: perform cleaning operations on the candidate data items using a cleaning execution agent to generate standardized data.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images