Real-time dynamic cleaning and visual analysis method for multi-source heterogeneous data

Abstract

The invention discloses a real-time dynamic cleaning and visual analysis method for multi-source heterogeneous data, and relates to the technical field of data analysis. Multi-source data access classification: converting formats through an adapter and performing classification labeling; a real-time stream processing framework composed of Kafka and Flink is constructed to realize low-delay processing; a multi-level rule engine is adopted to clean data, and abnormity is detected from predefined rules to machine learning; establishing a multi-dimensional system to evaluate data quality, and analyzing and fusing heterogeneous data by using entities; a visualization mode is automatically selected according to data features, parameters are dynamically adjusted, an interactive analysis interface is provided, and a strategy is optimized through reinforcement learning. According to the method, efficient cleaning and real-time analysis of multi-source heterogeneous data are realized, and the data processing efficiency and quality are improved; a visualization mode is automatically matched, the operation difficulty is reduced, and the user analysis efficiency is improved; interaction experience is optimized through reinforcement learning, and in combination with security protection and scheduling, data security is guaranteed, and powerful support is provided for decision making.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and in particular to a real-time dynamic cleaning and visual analysis method for multi-source heterogeneous data. BACKGROUND

[0002] In the current wave of digitalization sweeping the globe, data, as the core asset of enterprise and organization development, is showing unprecedented diversification and complexity in its sources. In daily operations, data not only comes from traditional relational databases, but also includes real-time data streams generated by Internet of Things devices, text information from social media, log files, and other semi-structured and unstructured data. These multi-source heterogeneous data differ greatly in format, structure, and update frequency. Traditional data cleaning and analysis methods have exposed many limitations that are difficult to overcome when dealing with such data.

[0003] Traditional data cleaning mostly uses static rules and batch processing mode, which is difficult to adapt to the dynamic characteristics of data. For example, when processing transaction data on e-commerce platforms, fixed cleaning rules cannot timely respond to temporary adjustments in data format during promotional activities, resulting in a large number of abnormal data that cannot be effectively processed. The batch processing method requires data to accumulate to a certain size before processing, which makes data cleaning have a serious lag and cannot meet the needs of real-time business analysis and decision-making. In addition, traditional methods rely on manual setting of cleaning rules, and in the face of complex and variable data patterns, the cost of updating and maintaining the rules is extremely high, and it is difficult to identify and process implicit abnormal data, making it difficult to ensure data quality.

[0004] Existing visualization analysis tools also have obvious shortcomings. Most tools lack intelligence and cannot automatically identify data characteristics and match appropriate visualization methods. For example, when displaying high-dimensional data, if traditional column charts or line charts are still used, it will be difficult to intuitively understand the data information, reducing the value of visualization. In terms of interactive analysis, tools often cannot dynamically optimize according to user operation habits and analysis needs, users need to frequently manually adjust parameters, the operation process is cumbersome, and it is difficult to obtain analysis results that meet their own needs. At the same time, with the increasing strictness of data security and privacy protection regulations, traditional data processing systems lack the ability to protect sensitive information and trace data operations, and cannot effectively respond to data leakage risks and compliance challenges.

[0005] With the deepening of enterprise digital transformation, the data size continues to grow exponentially, and the real-time, accuracy, and security requirements of business scenarios for data processing are becoming higher and higher. Traditional data cleaning and visualization analysis technology has been unable to meet the needs of enterprises to mine data value and improve decision-making efficiency, and there is an urgent need for a method that can adapt to the characteristics of multi-source heterogeneous data, realize real-time dynamic cleaning and intelligent visualization analysis, to promote the innovative development of enterprises in the data-driven era. SUMMARY

[0006] The present invention proposes a real-time dynamic cleaning and visual analysis method for multi-source heterogeneous data to solve the problems mentioned in the above-mentioned prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a real-time dynamic cleaning and visual analysis method for multi-source heterogeneous data, comprising:

[0008] Data access and classification: Supports access to structured databases, semi-structured data, and unstructured text data; converts data from different sources into a unified format through data interface adapters, and classifies and labels them according to data type, source, and update frequency to form an initial data classification map;

[0009] Real-time data stream processing framework construction: Build a message queue based on Apache Kafka to achieve real-time data reception and distribution; use Apache Flink as the stream processing engine;

[0010] Dynamic data cleaning strategy development: Design a hierarchical data cleaning rule engine. The first layer cleans erroneous data based on predefined rules; the second layer identifies outliers through statistical analysis; the third layer uses machine learning models to detect hidden anomalies, and the cleaning rules are dynamically adjusted based on real-time data feedback.

[0011] Data quality assessment and monitoring: Establish a data quality assessment system and conduct quantitative assessments based on four dimensions: completeness, accuracy, consistency, and timeliness. The calculation formula is: Q = w1C + w2A + w3Co + w4T, where Q is the overall quality score, C is the completeness score, A is the accuracy score, Co is the consistency score, and T is the timeliness score. w1, w2, w3, and w4 are corresponding weights, with default values ​​of 0.3, 0.3, 0.2, and 0.2, respectively. Monitor data quality fluctuations in real time and trigger an alert when the score falls below the threshold.

[0012] Heterogeneous data fusion and association analysis: Use entity resolution technology to identify the same entity in different data sources and build an entity relationship graph; store entities and relationships through a graph database to support association queries; use a graph embedding algorithm to map entities to a low-dimensional vector space. The calculation formula is: i =f(e i ,N(e i )), where vi is the embedding vector of entity ei, N(ei) is the set of neighbor nodes of entity ei, and f is the graph embedding function;

[0013] Adaptive visualization modeling: automatically select visualization methods based on data characteristics, build a visualization template library; for time series data, prefer to select line charts; for high-dimensional data, use parallel coordinates or heat maps; for hierarchical data, use tree charts or sunburst charts; the legends and coordinate axis ranges of visualization parameters are dynamically adjusted according to data distribution, and the adjustment algorithm is: P = P0 + a x AD, where P is the adjusted parameter value, P0 is the initial parameter value, a is the adjustment coefficient, and AD is the data distribution change amount;

[0014] Interactive analysis and feedback optimization: provide a drag-and-drop query interface, support user-defined analysis dimensions and indicators; user operations generate analysis logs, learn user preferences through reinforcement learning algorithms, and optimize visualization recommendation strategies; the system dynamically adjusts data cleaning rules and analysis models according to user feedback to form a closed-loop optimization mechanism.

[0015] Further, it further comprises:

[0016] Abnormal data traceability and repair: Establish an abnormal data traceability path to record the entire data cleaning process; for abnormal data cleaned out, repair solutions are found through association rule mining, and the repair success rate calculation formula is: Where R is the repair success rate, N repaired is the amount of successfully repaired abnormal data, and N anomaly is the total amount of abnormal data; when the repair success rate is lower than the threshold, the abnormal data is marked as to be manually processed.

[0017] Data security and privacy protection: sensitive information identification in the data access stage, differential privacy processing of sensitive data using data desensitization algorithm; the privacy budget allocation formula of the differential privacy protection mechanism is: Where ε total is the total privacy budget, and ε i is the privacy budget allocation value of the i th processing link.

[0018] Further, in the dynamic data cleaning strategy development step, the parameter update of the machine learning model adopts an online learning algorithm, and the weight update formula is: Where wt is the weight parameter of the t th iteration, η is the learning rate, is the loss function gradient, xt and yt are the t th training sample and label; the influence of historical data on the model is controlled through a sliding window mechanism, and the window size w is dynamically adjusted according to the data concept drift speed.

[0019] Further, in the data quality evaluation and monitoring step, the calculation of the timeliness score T uses an exponential decay model: T = e -λ×Δt, where λ is the decay coefficient and Δt is the data update time interval; differentiated evaluation of timeliness is achieved by setting the decay coefficient threshold for different business scenarios.

[0020] Furthermore, during the heterogeneous data fusion and association analysis steps, entity resolution uses a deep learning-based matching algorithm. The matching score is calculated as follows: S = σ(W [v1; v2] + b), where S is the matching score, σ is the activation function, W and b are learnable parameters, and v1 and v2 are the feature vectors of the entities to be matched. When S exceeds a threshold, the entities are considered to be the same.

[0021] Furthermore, in the adaptive visualization modeling step, the visualization template selects a multi-armed bandit algorithm to dynamically adjust the recommendation strategy based on historical click-through rates and user dwell time. The exploration-exploitation balance formula is: Where at is the visualization template selected for the tth time, Q t (a) is the historical average reward of template a, N t (a) is the number of times template a is selected, and c is the exploration coefficient.

[0022] Furthermore, in the interactive analysis and feedback optimization step, the reinforcement learning algorithm adopts the deep Q network DQN, and the Q value update formula is: Q(s t ,a t )←Q(s t ,a t )+α[r t +γmax a Q(s t+1 ,a)-Q(s t ,a t )] Where st is the state, at is the action, rt is the reward, α is the learning rate, and γ is the discount factor. The experience replay mechanism is used to improve training stability.

[0023] Furthermore, the system adopts a microservice architecture, each functional module communicates through the API gateway, the data processing process supports dynamic orchestration, and the flexible configuration and scheduling of processing nodes are achieved through the workflow engine Camunda.

[0024] Furthermore, the system supports distributed deployment, with a single cluster extending to 100 nodes. It uses an adaptive resource scheduling algorithm to dynamically allocate computing resources based on data traffic and processing load. The resource utilization calculation formula is: , where U is the resource utilization rate, ui is the resource utilization rate of the i-th node, and wi is the weight of the i-th node; the stability of the system under peak traffic is achieved through the automatic expansion and contraction mechanism.

[0025] Compared with the existing technology, the beneficial effects of the present invention are:

[0026] In the data cleaning link, the multi-level dynamic cleaning strategy significantly enhances the system's processing capacity for complex data. The combination of predefined rules, statistical analysis and machine learning algorithms can quickly and accurately identify and process various types of abnormal data, whether it is obvious format errors or hidden outliers. At the same time, the dynamic adjustment mechanism of cleaning rules enables the system to automatically adapt to data changes, ensuring the accuracy and timeliness of data cleaning, and providing a high-quality data foundation for subsequent analysis.

[0027] The real-time data stream processing framework is based on advanced ApacheKafka and ApacheFlink technologies, achieving low-latency and high-concurrency processing of data, and completely solving the lag problem of traditional batch processing mode. Whether it is to deal with daily operational data of enterprises or sudden large-scale data traffic, the system can quickly respond to meet the needs of real-time business analysis and decision-making.

[0028] The adaptive visualization modeling function is a highlight of the method. The system can automatically select the most suitable visualization method according to the data characteristics and dynamically adjust the visualization parameters to present complex data in intuitive and easy-to-understand charts. Users do not need to manually perform tedious settings to obtain clear and effective data display, greatly reducing the threshold of data understanding and improving the efficiency and effectiveness of data visualization.

[0029] The interactive analysis and feedback optimization mechanism further improves user experience. By learning the user's operation habits and analysis preferences, the system can dynamically adjust the analysis strategy and visualization recommendations to provide personalized analysis services for users. Users can more conveniently explore and analyze data in depth, improving the scientificity and accuracy of decision-making.

[0030] In terms of data security and privacy protection, the method takes strict protection measures from data access to processing, sensitive information identification and data desensitization technology effectively prevent data leakage, and blockchain technology ensures the traceability of data processing process, provides solid guarantee for enterprise data security. In addition, the micro-service architecture and distributed deployment give the system strong scalability and flexibility, which can easily cope with changes in data size and business demand, and provide strong support for the digital development of enterprises. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The schematic diagram of the real-time dynamic cleaning and visualization analysis method of multi-source heterogeneous data proposed by the present application;

[0032] Figure 2 The data cleaning efficiency comparison schematic diagram of the real-time dynamic cleaning and visualization analysis method of multi-source heterogeneous data proposed by the present application;

[0033] Figure 3 The abnormal data detection rate diagram of the real-time dynamic cleaning and visual analysis method of multi-source heterogeneous data proposed in the application is shown. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the application will be apparently and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the application.

[0035] In the description of the application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the application.

[0036] In addition, the terms "first", "second" are only for descriptive purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited. In addition, the terms "mounting", "connecting", "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through an intermediate medium, or the communication between two elements. For a person of ordinary skill in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances, and the application will be further described in detail below with reference to the drawings.

[0037] REFERENCE Figures 1 to 3 : A specific embodiment of a real-time dynamic cleaning and visual analysis method of multi-source heterogeneous data

[0038] 1. Multi-source data access and classification

[0039] The system adopts a plug-in architecture to design the data access layer, and adapts different data sources through standardized interfaces. For structured databases such as MySQL and Oracle, JDBC drivers are used to establish connections, and data is obtained through pagination queries, with a single query data volume controlled within 100,000 to avoid memory overflow. For semi-structured XML and JSON data, custom parsers are used to extract key fields, and regular expressions are used to match data patterns to convert them into a unified JSON format. Unstructured text data such as log files and social media comments are collected through the Nutch crawler framework, with the collection frequency dynamically adjusted according to the data source update speed, with a minimum interval of 5 minutes.

[0040] In the data classification stage, a classification model based on decision tree is constructed. The model input includes 12 feature dimensions such as data format type (structured, semi-structured, unstructured), data source label (e-commerce platform, social media, Internet of Things device), and data update frequency (real-time, minute-level, hour-level, etc.). The decision tree is constructed using the C4.5 algorithm, and the optimal partition attribute is selected through information gain rate to classify data into 8 categories such as transaction data, user behavior data, and device monitoring data. After classification, the system automatically generates labels for each category of data and stores them in the metadata management module, facilitating subsequent data processing and retrieval.

[0041] 2. Real-time data stream processing framework construction

[0042] A distributed message queue is built based on Apache Kafka, with 3 Broker nodes each configured with 8GB of memory and 4 cores of CPU. Kafka topics are divided by data category, such as the "transaction-data" topic for storing transaction data, with 16 partitions set for each topic to improve concurrent processing capability. The Flink cluster is deployed in Standalone mode, including 1 JobManager node and 4 TaskManager nodes, each TaskManager configured with 2 Slots, with a single node capable of processing 100,000 pieces of data per second.

[0043] The topology of Flink job adopts Source-Transform-Sink mode. The Source part uses KafkaConsumer to read data from the corresponding topic, sets the automatic commit offset interval to 5 seconds to prevent data duplication. In the Transform stage, multiple Operators are combined according to data cleaning and analysis requirements, such as MapOperator for data format conversion and FilterOperator for preliminary data filtering. The Sink part writes the processed data to the HBase database for storage or pushes it to the visualization module for display. To ensure low latency in data processing, the system sets the Checkpoint interval to 1000 milliseconds to ensure that the processing progress can be quickly restored in the event of a failure, with a recovery time target of within 30 seconds.

[0044] 3. Dynamic data cleaning strategy

[0045] The data cleaning rule engine implements multi-level cleaning using rule chains. The first layer of pre-defined rule library contains more than 200 basic rules, such as setting the rule "amount must be greater than 0 and less than 1000000" for the amount field in transaction data, and checking the format of the date field to ensure it conforms to "YYYY-MM-DD". The rule execution uses the Rete algorithm, which can handle 50,000 rule matching requests per second. The second layer of statistical analysis module uses the 3σ criterion to identify outliers for numerical data. The calculation formula is: where μ is the data mean, σ is the standard deviation, n is the data sample size, and xi is the ith data value. When the data point is outside the range [μ-3σ, μ+3σ], it is determined to be an outlier.

[0046] The third layer uses the Isolation Forest algorithm to detect implicit anomalies. The model training sets the number of trees to 100 and the sub-sample size to 256. The system evaluates the cleaning rules every hour and dynamically adjusts the rule priority and parameters based on the data cleaning effect (such as outlier detection rate, false positive rate). For example, if the false positive rate of a rule exceeds 10% for three consecutive times, its execution priority will be reduced; if the anomaly detection rate of a certain type of data is lower than expected, related rules will be automatically added or parameters will be adjusted.

[0047] 4. Data quality evaluation and monitoring

[0048] The data quality evaluation system is executed by a Python script at regular intervals (default every hour) and quantitatively scores from four dimensions: integrity, accuracy, consistency, and timeliness. The integrity score C is calculated by the proportion of non-empty fields, with the formula:

[0049] Accuracy score A is calculated by comparing the cleaned data with authoritative data sources, such as matching transaction data with bank statements. Consistency score Co is used to evaluate the inconsistency between multi-source data, calculated by the proportion of conflicting records. Timeliness score T uses an exponential decay model: T = e -λ×Δt where λ is the decay coefficient (set according to the importance of the data, default 0.1), and Δt is the data update interval (in hours).

[0050] The system uses Grafana to build a monitoring panel to display data quality scores and trends in each dimension in real time. When the comprehensive score Q is less than 0.7, trigger email and SMS dual alarm, alarm information contains specific data quality problems and impact range. Operation and maintenance personnel can quickly locate the problem data source and processing link through the monitoring interface for targeted optimization.

[0051] 5. Heterogeneous data fusion and correlation analysis

[0052] Entity resolution uses a deep learning-based twin network model. The model input is the feature vector of two entities to be matched, and the output is the matching probability. Training data is extracted from 100,000 pairs of entities (positive and negative sample ratio 1:1) from historical data, trained using the Adam optimizer with a learning rate of 0.001 and a training period of 50 epochs. The matching score calculation formula is: S = σ(W·[v1;v2]+b), where S is the matching score, σ is the Sigmoid activation function, W and b are learnable parameters, and [v1;v2] represents concatenating the two entity feature vectors. When S exceeds 0.8, it is determined to be the same entity.

[0053] Entity relationship graph construction uses Neo4j graph database, nodes represent entities (such as users, goods, orders), edges represent relationships between entities (such as purchase, browse, comment). The system updates the graph regularly (every morning) and performs batch import operations through Cypher statements. Correlation analysis supports complex queries, such as querying "which other goods did users who purchased A goods also purchase", which is implemented through graph traversal algorithms (such as breadth-first search), with an average query response time of less than 200 milliseconds.

[0054] 6. Adaptive visual modeling visualization

[0055] The template library contains more than 30 visualization types, each template corresponds to different data features and analysis scenarios. The system first extracts features from the data, including data dimensions (single dimension, multi-dimension), data types (numeric, categorical), data distribution (normal distribution, skew distribution), etc. According to the feature matching visualization template, for example, for single-dimensional numeric data with time series distribution, prefer to use line chart; for multi-dimensional categorical data, recommend using Sankey diagram.

[0056] The dynamic adjustment of visualization parameters is based on data statistical information. Taking a line chart as an example, the coordinate axis range is automatically adjusted according to the minimum and maximum values of the data, and the formula is:

[0057] X-axis range = [min(x) - δ, max(x) + δ]

[0058] Y-axis range = [min(y) - δ, max(y) + δ];

[0059] where δ is the buffer value (set to 5% of the data range), x and y are the horizontal and vertical coordinate values of the data, respectively. The system supports user manual fine-tuning of visualization parameters, and the adjusted parameters are recorded and used to optimize subsequent recommendation strategies.

[0060] 7. Interactive analysis and feedback optimization

[0061] The system provides a web-based drag-and-drop query interface, and users generate analysis requests by dragging data fields to different analysis areas (such as dimension area, indicator area, and filtering area). The request parsing module converts user operations into SQL or Cypher query statements and sends them to the data source for querying. After the query result is returned, it is visualized through the ECharts chart library.

[0062] User operation behaviors (such as clicking, dragging, and dwell time) are recorded as logs, and log data includes operation time, operation type, and involved data fields. The reinforcement learning module learns user preferences using a deep Q network (DQN), with a state space containing 10 feature dimensions such as current visualization chart type and user operation history, and an action space containing 8 operations such as switching chart types and adjusting parameters. The Q value update formula is: Q(s t ,a t )←Q(s t ,a t )+α[r t +γmax a Q(s t+1 ,a)-Q(s t ,a t )] where st is the state, at is the action, rt is the reward (a reward value of -1 to 1 is given according to user dwell time and subsequent operations), α is the learning rate (0.1), and γ is the discount factor (0.9). The system performs model training every 100 operation logs collected, continuously optimizing visualization recommendation and analysis strategies.

[0063] 8. Abnormal data tracing and repair

[0064] The abnormal data traceability module processes logs for each data record during data cleaning, including data source, cleaning rule execution record, processing time, and other information. When abnormal data is found, a traceability path is constructed by tracing the logs. For example, for a transaction data marked as abnormal, the data source table and the specific rule triggered during cleaning (such as amount out of range) can be traced back.

[0065] Abnormal data repair uses two strategies: for simple format errors (such as non-standard date format), automatic repair through rule matching; for complex logical errors (such as amount calculation error), use association rule mining to find possible repair solutions. Extract association rules from historical correct data, for example, "when the order amount is inconsistent with the product unit price and quantity, if the quantity is correct, repair the amount as unit price x quantity". The repair success rate calculation formula is: where R is the repair success rate, N repaired is the amount of successfully repaired abnormal data, and N anomaly is the total amount of abnormal data. When the repair success rate is less than 60%, the abnormal data is marked as pending manual processing and pushed to the data administrator for review.

[0066] 9. Data security and privacy protection

[0067] In the data access stage, sensitive information is identified through regular expressions and machine learning models. For text data, a pre-trained BERT model is used for entity recognition to detect sensitive information such as ID numbers and bank card numbers; for structured data, a list of sensitive fields (such as name and mobile phone number fields) is configured for identification. The identified sensitive data is desensitized using k-anonymity algorithm, for example, replacing the user's name with "user" + random number, ensuring that at least k records in the same group of data have the same quasi-identifier.

[0068] The differential privacy protection mechanism is applied in the data statistical analysis link, and the privacy budget ∈ total is allocated according to the operation sensitivity. For query operations, a lower privacy budget (default 0.2) is allocated; for high-risk operations such as data mining, a higher budget (0.5) is allocated. The budget allocation formula is: where ∈ total is the total privacy budget, and ∈ i is the privacy budget allocation value for the i-th processing link. Through blockchain technology (using Hyperledger Fabric framework), the entire data processing process is recorded, and each processing node packages the operation record into a block to ensure that data operations are traceable and tamper-proof.

[0069] 10. System architecture and resource management

[0070] The system adopts a microservice architecture and is divided into eight core microservices, including data access service, cleaning service, analysis service, and visualization service. Each service communicates through an API gateway (using SpringCloudGateway). The API gateway implements unified identity authentication, traffic control, and routing forwarding functions, and the call delay between services is controlled within 10 ms. Service registration and discovery use the Eureka component to ensure dynamic scaling and failover of services.

[0071] The data processing flow is arranged through the Camunda workflow engine. Users can drag and drop processing nodes (such as data cleaning nodes and analysis nodes) in the visualization interface and configure the dependency relationships and parameters between nodes. The system uses a distributed deployment method and can be expanded to 100 nodes per cluster. The resource scheduling module is implemented based on YARN and uses an adaptive scheduling algorithm to dynamically allocate computing resources based on data traffic and processing load. The resource utilization rate calculation formula is: When the cluster resource utilization rate exceeds 80%, the automatic expansion mechanism is triggered to add new nodes from the resource pool; when the utilization rate is less than 30%, the system is scaled down to save resources.

[0072] The actual effect of the method is shown in the following table by comparing it with the traditional data processing method in the actual application of a certain e-commerce enterprise:

[0073] Evaluation index Traditional way Method of the present application Data cleaning time (10 million pieces) 30 minutes 2 minutes Abnormal data detection rate 75％ 95％ Data visualization setting time Average 15 minutes Automatic generation User operation efficiency improvement - 40％ Data leakage risk level High Low System resource utilization rate 50％ 85％

[0074] In terms of data cleaning, the method significantly reduces the cleaning time of 1 million data from 30 minutes to 2 minutes, and improves the abnormal data detection rate from 75% to 95% with the multi-level dynamic cleaning strategy and real-time stream processing framework, effectively ensuring data quality. The adaptive visualization modeling function automatically generates charts without manual user settings, saving a lot of time compared to traditional methods. The interactive analysis and feedback optimization mechanism improves user operation efficiency by 40%, allowing users to quickly obtain valuable information.

[0075] The data security and privacy protection measures significantly reduce the risk of data leakage, from high-risk level to low-risk level, meeting the compliance requirements of enterprises. The system architecture design and resource management strategy improve the resource utilization rate from 50% to 85%, ensuring processing performance while achieving efficient use of resources. Overall, the method surpasses traditional methods in terms of data processing efficiency, quality, security, and resource management, and has important application value and promotion significance.

[0076] The above merely describes preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes within the technical scope disclosed by the present application and according to the technical solutions and inventive concept of the present application, which should be covered within the protection scope of the present application.

Claims

1. A real-time dynamic cleaning and visual analysis method of multi-source heterogeneous data, characterized in that, Comprise: Data access and classification: support structured database, semi-structured data, unstructured text data access; Convert different source data into a unified format through a data interface adapter, classify and label according to data type, source and update frequency, and form an initial data classification atlas; Real-time data stream processing framework construction: build a message queue based on Apache Kafka to realize real-time receiving and distribution of data; use Apache Flink as a stream processing engine; Dynamic data cleaning strategy: design a hierarchical data cleaning rule engine, the first layer cleans error data based on predefined rules; The second layer identifies outliers through statistical analysis; the third layer uses machine learning models to detect implicit anomalies, and the cleaning rules are dynamically adjusted in real time according to data feedback; Data quality evaluation and monitoring: Establish a data quality evaluation system, quantitatively evaluate from four dimensions of completeness, accuracy, consistency and timeliness, the calculation formula is: Q = w1C + w2A + w3Co + w4T, where Q is the comprehensive quality score, C is the completeness score, A is the accuracy score, Co is the consistency score, T is the timeliness score, w1, w2, w3, w4 are the corresponding weights, default 0.3, 0.3, 0.2, 0.2 respectively; Real-time monitoring of data quality fluctuations, triggering an early warning when the score is below the threshold; Heterogeneous data fusion and correlation analysis: entity resolution technology is used to identify the same entity in different data sources, and an entity relationship graph is constructed; the entity and relationship are stored in a graph database to support correlation query; graph embedding algorithm is used to map the entity to a low-dimensional vector space, and the formula is: v i = f(e i ,N(e i )), where vi is the embedding vector of entity ei, N(ei) is the neighbor node set of entity ei, and f is the graph embedding function.

2. The method for real-time dynamic cleaning and visual analysis of multi-source heterogeneous data according to claim 1, wherein, Also include: Adaptive visualization modeling: automatically select visualization methods based on data characteristics, and build a visualization template library; for time series data, prefer to use line charts; For high-dimensional data, use parallel coordinates or heat maps; for hierarchical data, use tree charts or sunburst charts; Visualization parameters such as legends and coordinate ranges are dynamically adjusted according to data distribution, and the adjustment algorithm is: P = P0 + α × △D, where P is the adjusted parameter value, P0 is the initial parameter value, α is the adjustment coefficient, and ΔD is the data distribution change amount; Interactive analysis and feedback optimization: Provide a drag-and-drop query interface that supports user-defined analysis dimensions and indicators; User operations generate analysis logs, and the system learns user preferences through reinforcement learning algorithms to optimize visualization recommendation strategies; The system dynamically adjusts data cleaning rules and analysis models based on user feedback to form a closed-loop optimization mechanism; Abnormal data traceability and repair: Establish an abnormal data traceability path to record the entire data cleaning process; For the cleaned abnormal data, the repair scheme is mined through association rules, and the repair success rate calculation formula is: Where R is the repair success rate, N repaired is the amount of successfully repaired abnormal data, N anomaly is the total amount of abnormal data; when the repair success rate is lower than the threshold, the abnormal data is marked as to be manually processed.

3. The method for real-time dynamic cleaning and visualized analysis of multi-source heterogeneous data according to claim 1, wherein, Also include: Data security and privacy protection: sensitive information identification is performed in the data access stage, and a data desensitization algorithm is used to process sensitive data; the privacy budget allocation formula of the differential privacy protection mechanism is: Wherein ε total is the total privacy budget, and ε i is the privacy budget allocation value of the i th processing link.

4. The method for real-time dynamic cleaning and visualized analysis of multi-source heterogeneous data according to claim 1, wherein, In the dynamic data cleaning policy making step, the parameter updating of the machine learning model adopts an online learning algorithm, and the weight updating formula is: wherein wt is the weight parameter of the tth iteration, η is the learning rate, is the gradient of the loss function, xt and yt are the tth training sample and label; the influence of historical data on the model is controlled through a sliding window mechanism, and the window size w is dynamically adjusted according to the data concept drift speed.

5. The method for real-time dynamic cleaning and visualized analysis of multi-source heterogeneous data according to claim 1, wherein, In the data quality evaluation and monitoring step, the calculation of the timeliness score T adopts an exponential decay model: T = e -λ×Δt where λ is the decay coefficient, and Δt is the data update time interval; and the differentiated evaluation of timeliness is realized by setting the decay coefficient threshold of different service scenarios.

6. The method for real-time dynamic cleaning and visualized analysis of multi-source heterogeneous data according to claim 1, wherein, In the step of heterogeneous data fusion and correlation analysis, the entity analysis uses a matching algorithm based on deep learning, and the matching score calculation formula is: S = σ (W·[v1;v2] + b), where S is the matching score, σ is the activation function, W and b are learnable parameters, and v1 and v2 are feature vectors of the entities to be matched. When S exceeds the threshold, it is determined to be the same entity.

7. The method for real-time dynamic cleaning and visualized analysis of multi-source heterogeneous data according to claim 2, wherein, In the adaptive visualization modeling step, the visualization template selection adopts a multi-armed bandit algorithm to dynamically adjust the recommendation strategy according to historical click rates and user dwell times, and the exploration-exploitation balance formula is: Where at is the visualization template selected for the tth time, Q t (a) is the historical average reward of template a, N t (a) is the number of selections of template a, and c is the exploration coefficient.

8. The method for real-time dynamic cleaning and visualized analysis of multi-source heterogeneous data according to claim 2, wherein, In the interactive analysis and feedback optimization step, the reinforcement learning algorithm adopts a deep Q network (DQN), and the Q value update formula is: Q(s t , a t )←Q(st,a t )+α[r t +γmax a Qs t+1 , a)-Q(s t , a t )] where st is the state, at is the action, rt is the reward, α is the learning rate, γ is the discount factor, and the training stability is improved through the experience replay mechanism.

9. The method for real-time dynamic cleansing and visualized analysis of multi-source heterogeneous data according to claim 1, wherein, The system uses a microservice architecture, and each functional module communicates through an API gateway. The data processing flow supports dynamic orchestration, and the processing node is flexibly configured and scheduled through the workflow engine Camunda.

10. The method for real-time dynamic cleansing and visualized analysis of multi-source heterogeneous data according to claim 1, wherein, The system supports distributed deployment, and a single cluster is extended to 100 nodes; an adaptive resource scheduling algorithm is adopted, data traffic and processing load are dynamically allocated to computing resources, and the resource utilization rate calculation formula is: Wherein U is the resource utilization rate, ui is the resource utilization rate of the ith node, and wi is the weight of the ith node; the stability of the system under peak traffic is realized through the automatic expansion and contraction mechanism.

Images