Adaptive multi-source heterogeneous data cleaning method and system based on large model driving

Abstract

The application provides a large model driven adaptive multi-source heterogeneous data cleaning method and system, which comprises an access layer, an edge layer, a regional layer and a core layer; the access layer accesses multi-source heterogeneous data and corresponding metadata, edge nodes of the edge layer deploy edge agents, preliminarily process the multi-source heterogeneous data, the regional layer deploys regional agents, is connected with a plurality of edge agents respectively and serves as a coordination center of the edge agents, the core layer is located in a central cloud and deploys a core agent to build a global cleaning knowledge graph; the edge agent cleans and filters data to generate a metadata report, the regional agent aggregates metadata and performs deep semantic analysis to dynamically generate a cleaning strategy, the core agent generates a global cleaning knowledge graph according to the cleaning strategy, performs task scheduling and distribution and issues a cleaning rule. The application distributes tasks to the nearest computing node through the three-layer agent collaborative architecture, improves the cleaning efficiency and has strong adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data management and computer technology, in particular to a self-adaptive multi-source heterogeneous data cleaning method and system based on large model driving. BACKGROUND

[0002] Data is the core asset in the digital era, but in practical applications, data often comes from multiple different business systems, databases, files or third-party APIs, forming the characteristics of "multi-source heterogeneity". In the governance of smart cities, multi-source heterogeneous data across fields and departments need to be unified and mined for value, such as traffic data, environmental data, security video data and municipal text data. However, these data sources have great differences in data structure, coding method, semantic expression and data quality, and data generally have quality problems such as different formats, content redundancy, information missing and logical contradictions.

[0003] In order to solve these problems, the industry has developed a variety of data cleaning technologies, which can be mainly summarized as follows:

[0004] Cleaning technology based on fixed rules: mainly relying on data engineers or business experts to manually write cleaning rules, such as using SQL statements, regular expressions, Python scripts, etc. This method is effective for structured and stable format data sources.

[0005] Cleaning technology based on statistics: this method uses statistical principles to identify and process data problems, for example, using Z-score or interquartile range to detect and process outliers, using mean, median or mode to fill in missing values, or using clustering algorithms to assist in identifying duplicate data.

[0006] Cleaning technology based on traditional machine learning: a classification model (such as SVM or decision tree) is trained to determine whether a record is "valid", or supervised learning is used to standardize data. However, this method usually requires a large amount of high-quality labeled training data, the cost of data labeling is high, and the model generalization ability is limited, making it difficult to adapt to new and unseen data formats.

[0007] Cleaning technology combined with language models: with the development of natural language processing (NLP) technology, some solutions have tried to use pre-trained language models for specific text cleaning tasks, such as address standardization or entity recognition. However, these applications are often single-task oriented and have not formed an end-to-end automated cleaning framework that is multi-source heterogeneous data-oriented, self-adaptive analysis and decision-making.

[0008] In addition, the existing technology still has the following shortcomings:

[0009] Traditional methods are difficult to understand the deep business semantics of data, cannot identify errors that violate common sense or business logic, resulting in incomplete cleaning and insufficient semantic understanding capabilities; most cleaning solutions require a large amount of manual intervention to define rules or annotate data, with limited automation level; existing methods are difficult to effectively process multi-source heterogeneous data, and cannot dynamically adjust the cleaning strategy according to the changes in the data environment, lacking generalization ability and adaptability; in a distributed scenario, centralized processing of all data will bring high communication cost and privacy risk, while simple local processing cannot achieve unified cleaning across sources, and cannot balance efficiency, quality and privacy.

[0010] To solve the above problems, the present application proposes a large model driven adaptive multi-source heterogeneous data cleaning method and system. SUMMARY

[0011] The present application proposes the following technical solutions to address one or more technical defects in the prior art.

[0012] Based on the first aspect of the present application, a large model driven adaptive multi-source heterogeneous data cleaning system is proposed, which is applied to the data governance construction of smart city, including an access layer, an edge layer, a regional layer and a core layer;

[0013] The access layer is used to access multi-source heterogeneous data and its corresponding metadata in the smart city management system, the edge nodes of the edge layer are deployed with edge agents, which are used to preliminarily process the multi-source heterogeneous data, the regional layer is deployed with regional agents, each regional agent is connected with multiple edge agents, serving as the coordination center of the edge agents, and the core layer is located in the central cloud, deployed with a core agent, used to build a global cleaning knowledge graph.

[0014] The edge agent cleans and filters the multi-source heterogeneous data of the access layer, generates a metadata report, and uploads the metadata report to the regional agent, the regional agent performs metadata aggregation and deep semantic analysis, dynamically generates a cleaning strategy, the core agent generates a global cleaning knowledge graph according to the cleaning strategy, performs task scheduling and distribution, and issues cleaning rules.

[0015] The present application effectively solves the deficiencies of traditional centralized data cleaning methods in efficiency and latency through a multi-layer agent system architecture, realizes distributed and layered cleaning and task scheduling of multi-source heterogeneous data, and greatly improves the overall processing capacity and adaptability of the system.

[0016] Further, the task scheduling and distribution of the core agent specifically includes:

[0017] The LLM large language model in the core intelligent agent performs deep semantic understanding on the task, dynamically evaluates the attribute vector of each to-be-assigned task according to semantic content and the metadata report, obtains the capability vector of each intelligent agent in real time, calculates the potential income of each task-agent pair, assigns each task to the intelligent agent with the maximum potential income, and outputs an optimal task assignment scheme.

[0018] This step realizes the optimal matching between the task and the intelligent agent, and significantly improves the intelligent level and resource utilization of the data cleaning task scheduling.

[0019] Further, the calculation formula of the potential income of the task-agent pair is:

[0020] ;

[0021] Among them, represents the potential income of the task-agent pair , , and respectively represent the cleaning quality function, the processing delay function and the privacy risk function of the task-agent pair , represents the weight vector driven by the LLM large language model, , represents the to-be-assigned task, , represents the capability vector of the intelligent agent, .

[0022] This step comprehensively considers the data cleaning quality, processing delay and privacy risk, realizes the quantitative optimization of task assignment, and enables the system to run efficiently and stably in a complex environment.

[0023] Further, for each task , it is assigned to the intelligent agent with the maximum potential income, and the calculation formula is:

[0024] ;

[0025] Among them, represents the intelligent agent with the maximum potential income, represents the to-be-assigned task, represents the potential income of the task-agent pair , represents the to-be-assigned task, , represents the capability vector of the intelligent agent, .

[0026] The application can achieve an optimal balance of cleaning efficiency and cleaning quality in a global range, ensuring that each cleaning task is performed by the most suitable agent.

[0027] Based on the second aspect of the application, a method for data cleaning based on the system described in any of the above is also proposed, which is applied to the data governance construction of smart city, comprising:

[0028] S1: Accessing multi-source heterogeneous data and its corresponding metadata in the smart city management system at the access layer, parsing the metadata and constructing a metadata knowledge graph;

[0029] S2: The edge agent of the edge layer performs preliminary cleaning and preprocessing on the multi-source heterogeneous data and generates a metadata report;

[0030] S3: The regional agent of the regional layer combines the metadata knowledge graph and the metadata report, performs cross-node data fusion and deep semantic analysis on the multi-source heterogeneous data processed by the multiple edge agents, and generates a cleaning strategy;

[0031] S4: The core agent combines the reinforcement learning mechanism of the LLM large language model to perform task scheduling and optimize the cleaning strategy.

[0032] Further, step S2 specifically includes that the edge agent converts the multi-source heterogeneous data into standardized internal representation through the parser and rule engine, eliminates redundant data in the multi-source heterogeneous data, filters invalid data, and fills in missing values, performs desensitization processing and classification filtering processing on the multi-source heterogeneous data through the LLM large language model, and encapsulates the processed multi-source heterogeneous data, processing log and local data quality into a metadata report.

[0033] Further, step S3 specifically includes that the regional agent uses the LLM large language model to perform cross-source entity recognition and disambiguation, aligns and merges data records pointing to the same business entity in the metadata report, eliminates data conflicts, and performs deep semantic analysis combined with the metadata knowledge graph, identifies complex event sequences, constructs a business entity relationship graph, and obtains a cleaning strategy.

[0034] This step can solve the conflict and redundancy problems between multi-source heterogeneous data, and improve the consistency and business availability of data.

[0035] Further, step S4 specifically comprises that the core agent monitors the environment of the LLM large language model, inputs the processed multi-source heterogeneous data into the LLM policy network, the LLM policy network outputs an optimal action instruction according to the semantic state of the multi-source heterogeneous data, the core agent receives the execution feedback of the optimal action instruction, the LLM large language model performs semantic analysis on the execution feedback and calculates a final reward, and the LLM policy network performs parameter updating and cleaning strategy optimization through a reinforcement learning algorithm in response to the received reward signal.

[0036] Further, the environment of the LLM large language model is composed of an edge agent, a regional agent, real-time data flow, a task queue and a system resource state.

[0037] The semantic state of the multi-source heterogeneous data includes structured data and unstructured data, the structured data includes CPU load and network delay, and the unstructured data includes a semantic analysis report of a current task of the LLM large language model.

[0038] This step enhances the perception and understanding ability of the system to the complex environment, and can improve the decision comprehensiveness and accuracy of the system.

[0039] Based on the third aspect of the present application, a computer program product is also proposed, which has one or more computer programs thereon, when the computer programs are executed by a computer processor, the method as described in any one of the above is implemented.

[0040] The technical effect of the present application is that the present application proposes an adaptive multi-source heterogeneous data cleaning method and system based on a large model driving, which intelligently allocates tasks to the nearest computing node through a three-layer agent collaborative architecture, significantly improves the efficiency of data cleaning, reduces the transmission amount and transmission delay of data, combines the LLM large language model to deeply understand the data semantics, identify and correct complex errors, improve the accuracy of data cleaning, and the continuous optimization mechanism of reinforcement learning enables the system to learn from experience, dynamically adapt to new data types, cleaning tasks and environmental changes, and the localization of the edge agent desensitization ensures data privacy from the source, which has a broad application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0041] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings.

[0042] Figure 1 FIG. 1 is a schematic diagram of an adaptive multi-source heterogeneous data cleaning system based on a large model driving according to an embodiment of the present application.

[0043] Figure 2is a flowchart of a large model driven adaptive multi-source heterogeneous data cleaning method provided by an embodiment of the application.

[0044] Figure 3 is a structural schematic diagram of a computer system of an electronic device suitable for implementing an embodiment of the application. DETAILED DESCRIPTION

[0045] The application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0046] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0047] Figure 1 A large model driven adaptive multi-source heterogeneous data cleaning system of the application is shown, which is applied to data governance construction of smart city including access layer 101, edge layer 102, regional layer 103 and core layer 104;

[0048] The access layer 101 is used to access multi-source heterogeneous data and its corresponding metadata in the smart city management system, the edge nodes of the edge layer 102 are deployed with edge agents, which are used to preliminarily process the multi-source heterogeneous data, the regional layer 103 is deployed with regional agents, each regional agent is connected with a plurality of edge agents, and the regional agent serves as a coordination center of the edge agent, and the core layer 104 is located in the central cloud and is deployed with a core agent, which is used to build a global cleaning knowledge graph.

[0049] The edge agent cleans and filters the multi-source heterogeneous data of the access layer 101, generates a metadata report, and uploads the metadata report to the regional agent, the regional agent performs metadata aggregation and deep semantic analysis, dynamically generates a cleaning strategy, the core agent generates a global cleaning knowledge graph according to the cleaning strategy, performs task scheduling and distribution, and issues a cleaning rule.

[0050] It should be noted that the task scheduling and distribution of the core agent specifically includes:

[0051] The LLM large language model in the core intelligent agent performs deep semantic understanding on the task, dynamically evaluates the attribute vector of each to-be-assigned task according to semantic content and the metadata report, obtains the capability vector of each intelligent agent in real time, calculates the potential income of each task-agent pair, assigns each task to an intelligent agent with maximum potential income, and outputs an optimal task assignment scheme.

[0052] It should be noted that the calculation formula of the potential income of the task-agent pair is:

[0053] ;

[0054] Among them, represents the potential income of the task-agent pair , , and respectively represent the cleaning quality function, the processing delay function and the privacy risk function of the task-agent pair , represents the weight vector driven by the LLM large language model, , represents the to-be-assigned task, , represents the capability vector of the intelligent agent, .

[0055] It should be noted that for each task , it is assigned to an intelligent agent with maximum potential income , and the calculation formula is:

[0056] ;

[0057] Among them, represents the intelligent agent with maximum potential income, represents the to-be-assigned task, represents the potential income of the task-agent pair , represents the to-be-assigned task, , represents the capability vector of the intelligent agent, .

[0058] It should be noted that the access layer accesses multiple source heterogeneous data (including structured databases, semi-structured log files and unstructured texts) and the corresponding metadata (including database structures, field annotations and API documents), and performs formatting, field extraction and standardization processing on the metadata to construct a unified metadata knowledge graph.

[0059] It should be noted that the edge layer deploys a lightweight edge agent on the edge node close to the data source, which combines traditional processing rules and lightweight pre-trained models to efficiently and locally clean and filter the original multi-source heterogeneous data, minimizes data transmission volume and transmission delay, and ensures data privacy from the source.

[0060] It should be noted that the regional data center of the regional layer deploys a regional agent, which uses a medium-scale LLM large language model with stronger computing power to deeply understand and process aggregated data.

[0061] It should be noted that the core agent is deployed in the central cloud, serving as the decision-making brain of the entire system, responsible for receiving the cleaning results of all regional agents, enabling efficient task coordination and scheduling with a global perspective, and performing LLM large language model-driven reinforcement learning optimization based thereon.

[0062] In specific embodiments, the core agent inputs a set of tasks to be allocated , a set of agent capabilities , and an LLM-driven weight vector , the LLM large language model dynamically evaluates the attribute vector of each task based on the semantic content and metadata of each task, including the size, complexity, request delay, and privacy risk of each task, and real-time acquisition of the capability vector of each agent , including the CPU computing processing capability, bandwidth communication capability, and efficiency capability of each agent, calculates the potential benefits for each task-agent pair, and allocates each task to the agent with the maximum potential benefit, outputting an optimal task allocation scheme .

[0063] It should be noted that the system distributes tasks intelligently to the nearest computing nodes through a three-layer agent architecture, reducing data transmission volume and transmission delay, improving data cleaning efficiency, and combining the deep understanding of data semantics by the LLM large language model to identify and correct complex errors that traditional methods cannot detect, improving data cleaning accuracy. The system also learns from experience and dynamically adapts to new data types, cleaning tasks, and environmental changes based on a reinforcement learning-based continuous optimization mechanism, and ensures data privacy at the source, especially in sensitive data fields. The application has significant advantages and broad application prospects in smart cities, industrial Internet of Things, financial risk control, and medical health fields, and can be used as a key component of enterprise-level data middle platform or big data platform, providing high-quality and reliable data foundation for various data-intensive applications.

[0064] Reference will now be made to Figure 2 ,Figure 2 A method for data cleaning based on the system according to any one of the preceding embodiments is shown, applied to the data governance construction of smart city, comprising:

[0065] S1: In the access layer, access the multi-source heterogeneous data in the smart city management system and the corresponding metadata, parse the metadata and construct the metadata knowledge graph;

[0066] S2: The edge agent of the edge layer performs preliminary cleaning and preprocessing on the multi-source heterogeneous data and generates a metadata report;

[0067] S3: The regional agent of the regional layer combines the metadata knowledge graph and the metadata report, performs cross-node data fusion and deep semantic analysis on the multi-source heterogeneous data processed by the multiple edge agents, and generates a cleaning strategy;

[0068] S4: The core agent combines the reinforcement learning mechanism of the LLM large language model to perform task scheduling and optimize the cleaning strategy.

[0069] It should be noted that step S2 specifically includes that the edge agent converts the multi-source heterogeneous data into standardized internal representation through the parser and rule engine, eliminates redundant data in the multi-source heterogeneous data, filters invalid data, and fills in missing values, performs desensitization processing and classification filtering processing on the multi-source heterogeneous data through the LLM large language model, and the edge agent encapsulates the processed multi-source heterogeneous data, processing log and local data quality into a metadata report.

[0070] It should be noted that the edge agent utilizes the locally deployed lightweight LLM large language model or professional model to perform preliminary semantic processing of sensitive data and complex data locally, identifies and desensitizes private information of text fields, and quickly classifies and filters unstructured data, and uploads the metadata report to the regional agent through a secure channel.

[0071] It should be noted that step S3 specifically includes that the regional agent utilizes the LLM large language model to perform cross-source entity recognition and disambiguation, aligns and merges data records pointing to the same business entity in the metadata report, eliminates data conflicts, and performs deep semantic analysis in combination with the metadata knowledge graph, identifies complex event sequences, constructs a business entity relationship graph, and obtains a cleaning strategy.

[0072] It should be noted that the regional agent analyzes the preliminary cleaned multi-source heterogeneous data and metadata reports uploaded from multiple edge agents in real time, and performs data stream aggregation, buffering and preliminary verification. When data conflicts are found, the LLM large language model can determine which data value is more reliable by combining metadata and business rules, or generate an optimal fused data value to complete data aggregation.

[0073] It should be noted that step S4 specifically includes that the core agent monitors the environment of the LLM large language model, inputs the processed multi-source heterogeneous data into the LLM policy network, the LLM policy network outputs the optimal action instruction according to the semantic state of the multi-source heterogeneous data, the core agent receives the execution feedback of the optimal action instruction, the LLM large language model performs semantic analysis on the execution feedback and calculates the final reward, and in response to the received reward signal, the LLM policy network updates parameters and optimizes cleaning strategies through a reinforcement learning algorithm.

[0074] It should be noted that the environment of the LLM large language model is composed of edge agents, regional agents, real-time data streams, task queues and system resource states.

[0075] The semantic state of the multi-source heterogeneous data includes structured data and unstructured data, the structured data includes CPU load and network delay, and the unstructured data includes semantic analysis reports of current tasks of the LLM large language model.

[0076] It should be noted that the core agent continuously optimizes task scheduling and cleaning rules using a reinforcement learning strategy optimization algorithm to achieve adaptive evolution of the system, and the LLM large language model plays the core roles of policy maker, knowledge injector and reward evaluator in this process.

[0077] In specific embodiments, the reinforcement learning strategy optimization algorithm performs reinforcement learning training cycles and adaptive evolution by constructing an LLM large language model driven reinforcement learning model, the LLM large language model driven reinforcement learning model includes a core agent, an environment of the LLM large language model, a semantic state of the multi-source heterogeneous data, actions and an LLM policy network, wherein the core agent is responsible for task allocation decision and cleaning rule issuance, the environment of the LLM large language model is composed of edge agents, regional agents, real-time data streams, task queues and system resource states.

[0078] The state is a comprehensive snapshot of the environment at a certain time, including structured data and unstructured data, the structured data includes CPU load and network delay, and the unstructured data includes semantic analysis reports of current tasks of the LLM large language model.

[0079] The action is a specific instruction set generated by the LLM large language model in a specific state, for example, "assign task T j to agent A i , and issue the cleaning rule R k generated by the large model";

[0080] The LLM strategy is a deep neural network based on a large-scale LLM large language model, which takes the semantic state of multi-source heterogeneous data as input and outputs the probability distribution of the optimal action.

[0081] The reinforcement learning model is trained, and the LLM strategy network analyzes the semantic state input of the multi-source heterogeneous data, for example, "CPU_load = 95% and queue_length = 500" numerical information is understood as "edge agent A faces serious computing bottleneck"; The LLM strategy network generates the optimal action according to the semantic state, which is not a simple ID or vector, but a specific executable instruction, and is accompanied by decision reason. After the action is executed, the core agent receives the feedback report of each layer of agent, the LLM large language model performs semantic analysis on the feedback report and calculates the final reward. If the report shows that the task is successfully completed and the local desensitization is reduced by 80%, the LLM large language model gives a high score reward and attaches an explanation: "the action effectively reduces the privacy risk and network cost";

[0082] The LLM strategy network updates the parameters according to the received reward signal, and in this process, the prior knowledge and domain understanding of the LLM large language model are injected into the LLM strategy network, so that the reinforcement learning model learns from experience and quickly generalizes to new data types and cleaning tasks. The system can adjust and optimize the scheduling strategy and cleaning rule in real time according to the changing data flow and system load, with high-level adaptability and generalization ability.

[0083] It should be noted that the present application proposes a large model driven adaptive multi-source heterogeneous data cleaning method and system, which intelligently distributes tasks to the nearest computing node through a three-layer agent collaborative architecture, significantly improving the efficiency of data cleaning, while reducing the transmission amount and transmission delay of data. Combined with the LLM large language model, it can deeply understand the data semantics, identify and correct complex errors, improve the accuracy of data cleaning, and the continuous optimization mechanism of reinforcement learning enables the system to learn from experience, dynamically adapt to new data types, cleaning tasks and environmental changes, and the localization of edge agent desensitization ensures data privacy from the source, with broad application prospects.

[0084] The following refers to Figure 3It shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0085] like Figure 3 As shown, the computer system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or programs loaded from storage section 308 into random access memory (RAM) 303. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0086] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a liquid crystal display (LCD) and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card and a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 310 as needed so that computer programs read from it can be installed into storage section 308 as needed.

[0087] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable storage medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above-described functions defined in the methods of the present application are performed. Note that the computer readable storage medium of the present application can be either a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be - but is not limited to - an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable storage medium that can be used for by or in connection with an instruction execution system, apparatus or device, and that can carry or contain program code for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable storage medium can be transmitted as program code signals using any suitable transmission medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the above.

[0088] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0089] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0090] The modules involved in the embodiments of the present application can be implemented in the form of software or hardware.

[0091] As another aspect, the application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: include an access layer, an edge layer, a region layer and a core layer; wherein the access layer is configured to access multi-source heterogeneous data and metadata corresponding to the multi-source heterogeneous data, the edge nodes of the edge layer are deployed with edge agents, and the edge agents are configured to perform preliminary processing on the multi-source heterogeneous data, the region layer is deployed with region agents, each region agent is connected with a plurality of edge agents, and each region agent serves as a coordination center of the edge agents, and the core layer is located in a central cloud and is deployed with a core agent, and the core agent is configured to construct a global cleaning knowledge graph; the edge agent cleans and filters the multi-source heterogeneous data of the access layer to generate a metadata report, and then uploads the metadata report to the region agent, the region agent performs metadata aggregation and deep semantic analysis, dynamically generates a cleaning strategy, and the core agent generates a global cleaning knowledge graph according to the cleaning strategy, performs task scheduling and distribution, and issues a cleaning rule.

[0092] Finally, it should be explained that the above description is only the preferred embodiments of the application and the explanation of the applied technical principles. Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the above inventive concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. An adaptive multi-source heterogeneous data cleaning system driven by a large model, applied to data governance construction in smart cities, characterized in that... It includes the access layer, edge layer, regional layer, and core layer; The access layer is used to access multi-source heterogeneous data and its corresponding metadata in the smart city management system. The edge nodes of the edge layer are deployed with edge agents for preliminary processing of the multi-source heterogeneous data. The regional layer is deployed with regional agents, each of which is connected to multiple edge agents and serves as the coordination center for the edge agents. The core layer is located in the central cloud and is deployed with core agents for building a global clean knowledge graph. The edge agent cleans and filters the multi-source heterogeneous data of the access layer, generates a metadata report, and then uploads the metadata report to the regional agent. The regional agent performs metadata aggregation and deep semantic parsing to dynamically generate a cleaning strategy. The core agent generates a global cleaning knowledge graph based on the cleaning strategy, performs task scheduling and allocation, and issues cleaning rules. The core intelligent agent performs task scheduling and allocation specifically including: The LLM large language model in the core agent performs deep semantic understanding of the task. The LLM large language model dynamically evaluates the attribute vector of each task to be assigned based on the semantic content and the metadata report, obtains the capability vector of each agent in real time, calculates the potential benefit of each task-agent pair, assigns each task to the agent that maximizes the potential benefit, and outputs the optimal task allocation scheme. The formula for calculating the potential reward of the task-agent pair is as follows: in, Representing task-agent pairs Potential benefits , and These represent task-agent pairs respectively. The cleaning quality function, processing delay function, and privacy risk function. This represents the weight vector driven by the LLM large language model. , Indicates tasks to be assigned. , A capability vector representing an agent. .

2. The system according to claim 1, characterized in that, For each task Assign it to the agent that maximizes potential gains. The calculation formula is: in, This represents an agent that maximizes potential gains. Indicates tasks to be assigned. Representing task-agent pairs Potential benefits Indicates tasks to be assigned. , A capability vector representing an agent. .

3. A method for data cleaning based on the system described in any one of claims 1-2, applied to the data governance construction of smart cities, characterized in that, include: S1: Access the multi-source heterogeneous data and its corresponding metadata in the smart city management system at the access layer, parse the metadata and construct a metadata knowledge graph; S2: The edge agent of the edge layer performs preliminary cleaning and preprocessing of the multi-source heterogeneous data and generates a metadata report; S3: The regional agent of the regional layer combines the metadata knowledge graph and the metadata report to perform cross-node data fusion and deep semantic analysis on the multi-source heterogeneous data processed by multiple edge agents, and generate a cleaning strategy; S4: The core intelligent agent combines the reinforcement learning mechanism of the LLM large language model to schedule tasks and optimize the cleaning strategy.

4. The method according to claim 3, characterized in that, Step S2 specifically includes the following steps: the edge agent converts the multi-source heterogeneous data into a standardized internal representation through a parser and a rule engine, removes redundant data from the multi-source heterogeneous data, filters invalid data and fills in missing values, performs desensitization and classification filtering on the multi-source heterogeneous data through an LLM large language model, and the edge agent encapsulates the processed multi-source heterogeneous data, processing logs and local data quality into a metadata report.

5. The method according to claim 3, characterized in that, Step S3 specifically includes the following steps: the regional intelligent agent uses the LLM large language model to perform cross-source entity recognition and disambiguation, aligns and merges data records in the metadata report that point to the same business entity to eliminate data conflicts, and combines the metadata knowledge graph to perform deep semantic analysis, identify complex event sequences, construct a business entity relationship graph, and obtain a cleaning strategy.

6. The method according to claim 3, characterized in that, Step S4 specifically includes the following steps: the core agent monitors the environment of the LLM large language model, inputs the processed multi-source heterogeneous data into the LLM policy network, the LLM policy network outputs the optimal action instruction based on the semanticized state of the multi-source heterogeneous data, the core agent receives the execution feedback of the optimal action instruction, the LLM large language model performs semantic analysis on the execution feedback and calculates the final reward, and in response to the received reward signal, the LLM policy network performs parameter updates and cleaning strategy optimization through reinforcement learning algorithms.

7. The method according to claim 6, characterized in that, The environment of the LLM large language model consists of edge agents, regional agents, real-time data streams, task queues, and system resource states. The semanticized state of the multi-source heterogeneous data includes structured data and unstructured data. The structured data includes CPU load and network latency, and the unstructured data includes the semantic analysis report of the current task of the LLM large language model.

8. A computer program product having one or more computer programs thereon, characterized in that, When the computer program is executed by a computer processor, the method described in any one of claims 3-7 is performed.

Images