Data processing system based on artificial intelligence algorithm

Abstract

The invention discloses a data processing system based on an artificial intelligence algorithm, and relates to the technical field of artificial intelligence and data processing, and the system comprises a data input interface which is used for receiving a multi-source heterogeneous data stream; and the preprocessing engine is connected with the data input interface and comprises a dynamic metadata sensing unit, an incremental quality evaluation unit, a bidirectional verification self-repairing unit and a closed-loop feedback optimization unit. According to the data processing system based on the artificial intelligence algorithm, through dynamic metadata perception and an incremental quality evaluation mechanism, the manual intervention requirement of a multi-source heterogeneous data preprocessing stage is reduced, and the problem of adaptation stiffness caused by the fact that a traditional method depends on a static rule is solved; by combining the bidirectional verification design of the business rule and the model feedback, the dual reliability of the data recovery strategy in logic rationality and algorithm compatibility is ensured, the risk of overfitting or scene mismatching caused by a single verification mechanism is avoided, and the input data quality and decision accuracy of a downstream artificial intelligence model are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and data processing technology, and in particular to a data processing system based on artificial intelligence algorithms. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, its application scenarios in finance, healthcare, industry, and other fields are becoming increasingly complex. Data processing, as a fundamental component supporting the efficient operation of algorithmic models, directly impacts the accuracy and real-time performance of intelligent decision-making. Currently, the data input for AI systems often involves the integration of heterogeneous data from multiple sources, such as structured databases, unstructured text, real-time sensor streams, and offline historical records. This data varies significantly in format, size, update frequency, and quality. Traditional data preprocessing methods rely on manually pre-set cleansing rules or static quality assessment models, making them difficult to dynamically adapt to the frequent changes in business scenarios. Especially in scenarios with high real-time requirements, the dynamic and diverse nature of data streams leads to a significant increase in the cost of manual intervention. Existing systems often require a significant amount of time for manual labeling, anomaly detection, and remediation rule adjustments before data fusion. This not only slows down data processing efficiency but can also cause data quality fluctuations due to delayed rule updates, reducing the reliability of input data for subsequent algorithmic models. Furthermore, most AI-based data processing systems focus on optimizing the model itself, such as increasing training speed or reducing parameter size, while overlooking the underlying constraints of the data preprocessing stage on algorithm performance. This fragmented design results in a lack of end-to-end automation in the data quality assessment and repair process, preventing the formation of a closed-loop optimization mechanism. When faced with sudden data source changes or the introduction of new data formats, existing systems are often unable to quickly identify data features and generate appropriate cleaning strategies, forcing them to rely on manual intervention, severely limiting the system's scalability and actual deployment efficiency. Summary of the Invention

[0003] (1) Technical problems solved

[0004] In response to the shortcomings of the existing technology, the present invention provides a data processing system based on artificial intelligence algorithms, which solves the problem of how to achieve automated quality assessment and dynamic repair of multi-source heterogeneous data, thereby reducing dependence on manual intervention and improving the adaptability of data processing processes to complex business scenarios.

[0005] (2) Technical solution

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A data processing system based on artificial intelligence algorithm, comprising:

[0007] Data input interface, used to receive multi-source heterogeneous data streams;

[0008] A preprocessing engine, connected to the data input interface, comprising a dynamic metadata perception unit, an incremental quality assessment unit, a bidirectional verification and self-repair unit, and a closed-loop feedback optimization unit;

[0009] The dynamic metadata perception unit is deployed on the edge computing node, and extracts the format features, temporal correlation features and semantic features of the data stream in real time through a self-supervised comparative learning framework to generate dynamic feature coding;

[0010] The incremental quality assessment unit constructs a dynamic weight index including the dimensions of integrity, consistency and timeliness based on the dynamic feature coding, and uses a sliding time window mechanism to update the index threshold in real time;

[0011] The bidirectional verification self-repair unit includes a rule verification subunit and a model feedback verification subunit, which respectively perform double verification on the repaired data based on the business constraint library of the knowledge graph and the output stability of the downstream model shadow copy;

[0012] The closed-loop feedback optimization unit generates an optimized repair decision path based on the verification results and downstream model performance feedback through a reinforcement learning strategy;

[0013] The output end of the preprocessing engine is connected to the algorithm model execution module, which is used to input the verified repair data into the downstream artificial intelligence algorithm.

[0014] During the operation of the data processing system, after multi-source heterogeneous data is connected to the system through the data input interface, it first enters the metadata perception stage of the preprocessing engine.

[0015] The dynamic metadata perception unit uses a lightweight feature extractor to perform real-time analysis and feature capture of the input data stream.

[0016] For structured data, the extractor identifies field names, data types, and numerical distribution patterns, and automatically detects missing fields or type misalignment issues; for unstructured text, it locates keyword conflicts and logical breakpoints through semantic context analysis; for real-time sensor data streams, the extractor simultaneously tracks timestamp sequences and numerical change trends, and identifies abnormal jitter or continuity interruptions.

[0017] When the data stream format mutates, the extractor performs a difference comparison based on the historical feature coding template. If a new field or format version upgrade is detected, it immediately triggers a feature coding template update notification and sends a format change identifier to the incremental quality assessment unit.

[0018] Preferably, the dynamic metadata perception unit includes:

[0019] A lightweight feature extractor uses parameter sharing technology to compress the network size and reduces memory usage through knowledge distillation, so that the feature extractor has low-latency response capabilities at edge nodes.

[0020] Preferably, the incremental quality assessment unit performs the following operations:

[0021] When a data source switch is detected, the evaluation threshold of the integrity dimension is automatically adjusted based on the gradient change of the historical quality indicator within the sliding time window. The integrity dimension includes the weighted calculation result of the field missing rate and the data coverage rate.

[0022] After receiving the dynamic feature code, the incremental quality assessment unit initiates the calculation of multi-dimensional quality indicators. The integrity dimension dynamically sets thresholds based on the historical distribution of field missing rate and data coverage. This includes: When a data source switch causes a sudden increase in field missing rate but an unsynchronized decrease in data coverage, this is considered a format compatibility issue, temporarily relaxing the integrity threshold and initiating cross-source consistency enhancement verification. The consistency dimension identifies naming conflicts or unit format mismatches by comparing the field mapping relationship between the current data stream and the associated data source. The timeliness dimension dynamically adjusts the timeout tolerance window for real-time streaming data based on the difference between the data generation timestamp and the processing delay.

[0023] When the quality indicator fluctuation within the sliding time window exceeds the preset sensitivity threshold, the evaluation unit automatically generates a quality trend report and triggers the threshold adaptive adjustment mechanism, including: if the fluctuation is caused by occasional anomalies, the threshold curve is smoothed using a sliding average algorithm; if the fluctuation shows a continuous upward trend, the root cause analysis module is activated to trace the data source or parser anomaly.

[0024] Preferably, the rule verification subunit includes:

[0025] The knowledge graph loading module calls the associated constraint subgraph according to the current business scenario type. The constraint subgraph contains causal association rules between data fields, numerical range boundary conditions, and cross-source data consistency rules;

[0026] The logic verification module performs multi-level matching on the field values ​​of the repaired data and the constraint subgraph, and marks the verification as failed if there is a conflict.

[0027] When the repaired data enters the two-way verification phase, the rule verification sub-unit calls the constraint subgraph in the knowledge graph that matches the current business scenario. For example, in a financial transaction scenario, a subgraph containing account flow continuity rules, transaction amount reasonableness boundaries, and anti-money laundering association conditions is loaded to perform a hierarchical verification of the repaired data. This includes first verifying whether the amount of a single transaction exceeds the account's historical transaction range, then verifying whether the chronological order of multiple transactions for the same account is logical, and finally cross-checking against blacklists in an external credit reporting system.

[0028] If the verification finds that the transaction amount is abnormal but the time sequence is reasonable, the dynamic threshold correction will be triggered and the reasonable range will be recalculated based on the account's transaction average over the past three months; if the time sequence conflicts and the associated account has a blacklist record, the batch of data will be frozen and a high-risk alert will be generated.

[0029] Preferably, the model feedback verification subunit includes:

[0030] Shadow copy generation module, which copies the structure and parameters of the downstream algorithm model;

[0031] The stability monitoring module compares the fluctuations in classification confidence, feature importance distribution offset, and prediction result variance when the data before and after repair are input into the shadow copy. If any of the indicators exceeds the preset tolerance threshold, the repair strategy is triggered to be regenerated. After the model feedback verification sub-unit inputs the repair data into the shadow copy, it monitors the stability changes of the model output, including: for image recognition tasks, statistically analyzing the similarity of the feature vectors of the images before and after repair and the difference in classification results; for time series prediction tasks, analyzing the degree of deviation of the smoothness of the prediction curve from the historical baseline. If the similarity drops by more than the tolerance or the prediction deviation continues to expand, the repair strategy is judged to be invalid, and it automatically rolls back to the previous valid version and marks the abnormal data source.

[0032] Preferably, the closed-loop feedback optimization unit includes:

[0033] The compound reward calculation module generates reward values ​​based on the verification pass rate of the repair operation, the change rate of the downstream model output standard deviation, and the resource consumption rate of the edge node;

[0034] The policy optimization module adopts a deep deterministic policy gradient algorithm to dynamically adjust the priority of the repair strategy according to the reward value, so that the repair accuracy and processing delay reach a Pareto optimal balance.

[0035] The closed-loop feedback optimization module dynamically optimizes repair strategies based on validation results and downstream model performance feedback. In real-time stream processing scenarios, the system prioritizes low latency and employs a fast repair strategy that skips deep validation of non-critical fields. In offline batch processing scenarios, a high-precision mode is enabled, allowing cross-batch data backtracking and multiple rounds of iterative repair.

[0036] When resource constraints lead to insufficient computing power at the edge node, the system automatically migrates non-real-time tasks to the cloud and retains the minimum necessary verification functions locally to ensure continuous processing of core data streams.

[0037] Preferably, a feature code cache is provided between the dynamic metadata perception unit and the incremental quality assessment unit for temporarily storing the dynamic feature code within the current time window and sending a trigger signal to the incremental quality assessment unit when the data source is switched.

[0038] Preferably, the knowledge graph loading module further includes:

[0039] The real-time update interface receives external input constraint rules and automatically expands the node and edge relationships of the constraint subgraph when a new business scenario is added.

[0040] When a new business scenario or data format is added, the system injects new rules through the knowledge graph update interface and performs a stress test during the first verification, including: extracting historical data to apply new rules. If the conflict rate increases abnormally, the rules are delayed and manual review is triggered. If the test passes, the knowledge graph node connection path is automatically optimized to improve the efficiency of subsequent verification.

[0041] For tasks migrated to the cloud for processing, the system records the complete metadata chain and processing log to ensure the traceability and consistency of local and cloud data, and automatically synchronizes the difference data after the network is restored.

[0042] Preferably, the data processing system based on artificial intelligence algorithm also includes:

[0043] A load balancing controller, connected to the dynamic metadata sensing unit, dynamically allocates data streams to idle nodes for feature extraction based on the real-time data traffic and CPU usage of each edge node. The load balancing controller monitors node load status in real time. When a node's processing delay approaches a threshold, it dynamically allocates data streams to idle nodes. During the migration process, the data blocks' temporal consistency and integrity checks are maintained to prevent data dislocation or loss due to network jitter.

[0044] Preferably, the load balancing controller performs the following operations: when the data flow of the edge node exceeds a preset peak value, redirecting part of the data flow to the adjacent node, and maintaining the temporal continuity of the dynamic feature coding during the redirection process.

[0045] Through the implementation of this solution, the system automates the entire process of processing heterogeneous multi-source data, from access, quality assessment, repair verification, to closed-loop optimization. This reduces the need for manual intervention while ensuring data processing accuracy and real-time performance in complex business scenarios. The collaborative mechanisms and adaptive strategy design between modules effectively address the problems of rigid rules, single-source validation, and limited scalability that plague traditional systems when merging dynamic data.

[0046] (3) Beneficial effects

[0047] The present invention provides a data processing system based on artificial intelligence algorithms. It has the following beneficial effects:

[0048] (1) This data processing system based on artificial intelligence algorithms reduces the need for manual intervention in the pre-processing stage of multi-source heterogeneous data through dynamic metadata perception and incremental quality assessment mechanisms, solving the adaptation rigidity problem caused by traditional methods relying on static rules; combining the two-way verification design of business rules and model feedback to ensure the dual reliability of data repair strategies in terms of logical rationality and algorithm compatibility, avoiding the risk of overfitting or scenario mismatch caused by a single verification mechanism, thereby improving the input data quality and decision accuracy of downstream artificial intelligence models.

[0049] (2) This data processing system based on artificial intelligence algorithms achieves adaptive processing capabilities in complex business scenarios through closed-loop feedback optimization and edge-cloud collaborative load balancing strategies, including: prioritizing low latency and high throughput in real-time streaming data scenarios, and supporting high-precision multi-round repair in offline batch processing; at the same time, the dynamically expanded knowledge graph constraint library and incremental learning mechanism effectively respond to sudden data source switching and new data format access, improving the system's scalability and deployment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0051] Figure 2 This is a control logic timing diagram of the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] See also Figure 1 and Figure 2 The present invention provides a technical solution: a data processing system based on artificial intelligence algorithm, comprising:

[0054] Data input interface, used to receive multi-source heterogeneous data streams;

[0055] The preprocessing engine is connected to the data input interface and includes a dynamic metadata perception unit, an incremental quality assessment unit, a bidirectional verification and self-repair unit, and a closed-loop feedback optimization unit;

[0056] The dynamic metadata perception unit is deployed on the edge computing node, and uses a self-supervised comparative learning framework to extract the format features, temporal correlation features, and semantic features of the data stream in real time to generate dynamic feature codes.

[0057] The incremental quality assessment unit builds dynamic weight indicators based on dynamic feature coding, including the dimensions of completeness, consistency, and timeliness, and uses a sliding time window mechanism to update the indicator threshold in real time;

[0058] The bidirectional verification self-repair unit includes a rule verification sub-unit and a model feedback verification sub-unit. These sub-units perform double verification on the repaired data based on the business constraint library of the knowledge graph and the output stability of the downstream model shadow copy.

[0059] The closed-loop feedback optimization unit uses a reinforcement learning strategy to generate an optimized repair decision path based on verification results and downstream model performance feedback;

[0060] The output of the preprocessing engine is connected to the algorithm model execution module, which is used to input the verified repair data into the downstream artificial intelligence algorithm.

[0061] The dynamic metadata perception unit includes:

[0062] The lightweight feature extractor uses parameter sharing technology to compress the network size and reduces memory usage through knowledge distillation, making the response delay of the feature extractor at the edge node less than 5 milliseconds.

[0063] It should be further explained that, in the specific implementation process, when the lightweight feature extractor runs on the edge node, it first merges the redundant convolution kernels and fully connected layer weights in the multi-layer neural network through parameter sharing technology to reduce the number of network parameters; at the same time, it adopts the knowledge distillation framework to migrate the feature mapping relationship in the pre-trained large metadata perception model to the lightweight model, retaining the key feature recognition capability while reducing memory usage.

[0064] When the data stream enters the edge node, the extractor parses the format and metadata of the data packet in real time, automatically identifies missing fields or type misalignment in structured data, detects semantic conflicts and context breaks in unstructured text, and triggers a dynamic sampling rate adjustment mechanism to balance processing accuracy and latency when anomalies in the timing correlation of real-time streaming data are detected.

[0065] In the scenario where edge computing resources are limited, if the memory usage of the current node exceeds the preset threshold, the extractor automatically switches to low-power mode, keeping only the core feature extraction layer running, and temporarily caches the segmented hash values ​​of the unprocessed data stream. After the resources are released, the feature coding generation of high-priority data is restored first.

[0066] In addition, when a sudden increase in the data flow rate causes the response delay to approach the upper limit of 5 milliseconds, the extractor dynamically allocates computing threads to idle cores based on historical load data counted in a sliding time window, and skips secondary verification of non-critical features, ensuring that sensor data with the highest real-time requirements is processed first.

[0067] The incremental quality assessment unit performs the following operations:

[0068] When a data source switch is detected, the evaluation threshold of the integrity dimension is automatically adjusted based on the gradient change of historical quality indicators within the sliding time window. The integrity dimension includes the weighted calculation results of the field missing rate and data coverage.

[0069] It should be further explained that, during the specific implementation process, when the system detects a data source switch or a data format update, the incremental quality assessment unit first obtains the changing trend of the historical quality indicators in the current sliding time window, and determines whether the data quality fluctuation is caused by source switching by calculating the gradient difference of the weighted values ​​of the field missing rate and data coverage in the integrity dimension in continuous time segments.

[0070] If the gradient difference exceeds the preset sensitivity threshold, the dynamic evaluation threshold adjustment mechanism will be activated, including: for data streams from structured databases, the tolerance upper limit of the field missing rate will be reset according to the field naming specifications and historical coverage distribution of the new data source; for unstructured text data streams, the evaluation weight of the timeliness dimension will be dynamically relaxed based on the semantic coherence detection results to avoid misjudgments caused by format parsing delays.

[0071] When the integrity indicator fluctuations tend to be stable within three consecutive time windows after the data source is switched, the original evaluation threshold is gradually restored, and the adjusted parameters are recorded in the historical knowledge base for rapid call in subsequent similar scenarios.

[0072] If a data format update causes a sudden increase in the field missing rate but the data coverage rate does not decrease synchronously, it is determined to be a format compatibility conflict, triggering the cross-source consistency rule enhancement verification. At the same time, the evaluation weight update of the timeliness dimension is temporarily frozen until the format parser completes adaptive calibration.

[0073] In addition, when historical data is insufficient or the gradient change is abnormally weak, a hybrid strategy combining the default threshold and the sliding average prediction value is adopted to prevent threshold drift problems caused by data sparsity, and a source switching quality assessment report is generated on the interface for review by operation and maintenance personnel.

[0074] The rule validation subunit includes:

[0075] The knowledge graph loading module calls the associated constraint subgraph based on the current business scenario type. The constraint subgraph contains causal association rules between data fields, numerical range boundary conditions, and cross-source data consistency rules.

[0076] The logic verification module performs multi-level matching between the field values ​​of the repaired data and the constraint subgraph. If there is a conflict, it is marked as a verification failure.

[0077] It should be further explained that in the specific implementation process, when the repaired data enters the rule verification sub-unit, the system loads the corresponding constraint sub-graph from the knowledge graph based on the current business scenario type. For example: in a medical scenario, a sub-graph containing the association rules between patient age and diagnosis results, drug dosage range boundaries, and cross-department data consistency conditions is called to perform multi-level verification on the field values ​​of the repaired data, including: first checking whether a single field violates the numerical boundary, such as blood pressure values ​​exceeding the physiological limit range, secondly verifying the cross-field logical relationship, such as the chronological matching of surgical records and anesthesia dosages, and finally cross-checking with external data sources, such as the compliance mapping of medical insurance codes and diagnosis and treatment items.

[0078] If verification at a certain level fails, differentiated processing will be performed based on the conflict type, including: for numerical out-of-bounds errors, automatically triggering the threshold dynamic correction mechanism, and redefining the temporary boundary based on historical data distribution; for logical association conflicts, tracing the data repair path and isolating the fields that cause the conflict, and generating manual review marks; for cross-source consistency inconsistencies, giving priority to freezing the data batch and initiating cross-system data source synchronization verification.

[0079] When a new business scenario is added, after injecting new constraint rules through the real-time update interface, the system automatically splits the general rules and scenario-specific rules in the original sub-graph, and performs compatibility tests on the new rules during the first verification, including: if the conflict rate between the repaired data and the new rules exceeds the preset threshold, the rule entry time is delayed and a rule conflict analysis report is triggered until the rule priority ranking is manually confirmed.

[0080] In addition, if complex conflicts are detected in multiple related fields within the same data batch during the verification process, the system will calculate the conflict weight based on the node connection density of the knowledge graph and prioritize resolving high-weight conflicts to reduce the risk of cascading errors.

[0081] The model feedback verification subunits include:

[0082] Shadow copy generation module, which copies the structure and parameters of the downstream algorithm model;

[0083] The stability monitoring module compares the classification confidence fluctuations, feature importance distribution shifts, and prediction result variances when data is input into the shadow copy before and after repair. If any indicator exceeds the preset tolerance threshold, the repair strategy is triggered to be regenerated.

[0084] It should be further explained that, during the specific implementation process, when the repair data input model feeds back to the verification sub-unit, the system first copies the structure and parameters of the downstream algorithm model to generate a shadow copy to ensure that its initial state is completely consistent with the production environment model.

[0085] The data before and after the repair are then input into the shadow copy respectively, and the classification confidence fluctuations output by the model are monitored. For example, if the difference in the highest category confidence of the same data sample before and after the repair exceeds the preset tolerance threshold, it is judged as a potential repair distortion; at the same time, the feature importance distribution shift is analyzed, and the standard deviation of the change in the weight of key features is calculated to identify abnormal model attention shifts caused by data repair.

[0086] For time series prediction tasks, the variance change rate of the prediction results is additionally counted. If the prediction variance of the repaired data exceeds the historical baseline range three times in a row, an early warning mechanism is triggered.

[0087] When any of the above indicators exceeds the limit, the system automatically traces back the repair operation steps that caused the exception. If the step involves field type conversion or missing value filling, the current repair strategy will be frozen first and rolled back to the previous valid version. If the exception is caused by a parsing error of a new data source, the data source will be temporarily blocked and the cross-source verification process will be started.

[0088] In financial risk control scenarios, if classification confidence fluctuations and feature importance shifts occur simultaneously but do not exceed the threshold, the system will correlate the changing trends of the two, including: when the direction of confidence fluctuations is negatively correlated with the direction of feature weight migration, it is judged that the model robustness has decreased, and the priority of the current repair strategy needs to be lowered and the verification time needs to be extended; if there is a positive correlation and the prediction variance is stable, the repair is considered effective but additional manual sampling and review are still required.

[0089] In addition, when some indicators in the same batch of data exceed the limit while other indicators are normal, the system performs differential processing based on the business risk level, including: for highly sensitive businesses, such as medical diagnosis, immediately interrupting the process and generating an exception report; for low-risk businesses, such as product recommendations, allowing data to enter the production model but marking the confidence level for downgraded use, and strengthening the verification weight of this scenario in subsequent feedback optimization.

[0090] The closed-loop feedback optimization unit includes:

[0091] The compound reward calculation module generates reward values ​​based on the verification pass rate of the repair operation, the change rate of the downstream model output standard deviation, and the resource consumption rate of the edge node;

[0092] The policy optimization module uses a deep deterministic policy gradient algorithm to dynamically adjust the priority of the repair strategy based on the reward value, so that the repair accuracy and processing delay reach a Pareto optimal balance.

[0093] It should be further explained that, during the specific implementation process, when the compound reward calculation module generates the reward value, it first dynamically allocates the weight coefficient of each indicator according to the preset business scenario type, including: increasing the weight ratio of processing delay in real-time sensitive scenarios, and strengthening the influencing factor of verification pass rate in accuracy sensitive scenarios.

[0094] For the verification pass rate of the repair operation, the proportion of data batches that passed the double verification in the current time window is counted, and the relative improvement is calculated based on the historical average pass rate; the downstream model performance improvement rate is determined by comparing the output stability when the same batch of data is input into the production model before and after the repair. If the highest confidence standard deviation in the classification task decreases or the prediction error fluctuation range in the regression task narrows, it is judged as a positive improvement.

[0095] The resource consumption rate is calculated by integrating the CPU usage, memory peak, and network transmission delay of the edge node. When the resource consumption of a node exceeds twice the average level of the cluster, a penalty coefficient is imposed on the repair operation initiated by the node.

[0096] When the policy optimization module uses a deep deterministic policy gradient algorithm, it dynamically adjusts the exploration and utilization balance of the repair strategy based on the real-time reward value. This includes: if the reward value shows an upward trend within three consecutive time windows, the random exploration ratio is reduced to stabilize the strategy; if the reward value fluctuation exceeds the historical variance threshold, the exploration action is increased to escape the local optimum.

[0097] When processing highly concurrent data streams, priority queues are set for different data types. For example, a low-latency repair strategy is used for real-time sensor streams, which prioritizes skipping non-critical field checks to maintain throughput. A high-precision repair mode is enabled for offline historical data, allowing backtracking across batches of data to complete missing fields.

[0098] If the system detects that the repair strategy of a certain type of data source is in a low reward state for a long time, it will trigger the strategy rollback mechanism, automatically switch to the historical optimal strategy version, and mark the data source feature code for manual review.

[0099] In addition, in resource-constrained edge nodes, when the compound reward value continues to decline due to insufficient computing resources, the strategy optimization module will forcibly downgrade some repair functions to cloud collaborative processing and retain the minimum necessary verification logic locally to maintain basic service quality.

[0100] A feature coding buffer is provided between the dynamic metadata perception unit and the incremental quality assessment unit for temporarily storing the dynamic feature coding within the current time window and sending a trigger signal to the incremental quality assessment unit when the data source is switched.

[0101] It should be further explained that, during the specific implementation process, when the feature code generated by the dynamic metadata perception unit is transmitted to the feature code cache, the system stores the code in segments according to the preset time window length, and records the generation timestamp and data source identifier of each segment.

[0102] If it is detected that the data source identifier changes within adjacent time windows, it is determined to be a normal data source switching event. The cache area immediately compresses all the codes in the current window into a snapshot file, attaches a source switching mark, and sends it to the incremental quality assessment unit; if mixed data source identifiers appear in the same time window and there is no clear switching signal, the burst switching detection mechanism is triggered, and the format feature differences in the encoding are preferentially extracted for conflict analysis, including: when the coexistence ratio of structured and unstructured data encoding exceeds the threshold, the cache area is automatically divided into multi-source independent storage blocks, and conflict marks are attached to each block for differentiated processing by the subsequent assessment unit.

[0103] When the edge node storage resources are insufficient, the cache area starts a dynamic cleanup strategy, including: for the encoding segments that have been sent to the evaluation unit and passed the preliminary verification, only the hash summary is retained and the original data space is released; for the encoding segments that have not been sent, low-priority content is selectively discarded according to the data priority label, and the metadata of the lost encoding is recorded in the log for traceability.

[0104] When the data source switching trigger signal is sent, if the incremental quality assessment unit feedback evaluation threshold adjustment fails, the cache will reload the encoding snapshot of the previous time window and perform feature similarity comparison with the current encoding. If the key feature matching degree is lower than the preset value, it is determined that the data source metadata is mislabeled, and the perception unit is triggered to re-extract the feature encoding.

[0105] In addition, in high-concurrency scenarios, if the encoding rate received by the cache area continuously exceeds the processing capacity of the evaluation unit, the traffic shaping mechanism is activated, including: enabling a direct channel to bypass the cache for real-time sensor data encoding and directly passing it to the evaluation unit for streaming processing, while reducing the sampling frequency of offline data encoding to balance the load.

[0106] The knowledge graph loading module further includes:

[0107] The interface is updated in real time. When a new business scenario is added, it receives external input constraint rules and automatically expands the node and edge relationships of the constraint subgraph.

[0108] It should be further explained that, during the specific implementation process, when a new business scenario is added, the system receives external input constraint rules through a real-time update interface, and first performs a syntax compliance check, including: if the rule contains undefined data fields or logical operator conflicts, the injection is rejected and an error report is generated; the rules that pass the check are broken down into atomic conditional units and automatically mapped to existing nodes in the knowledge graph according to the semantic association.

[0109] For newly added drug incompatibility rules in medical scenarios, the system identifies the drug ingredient nodes and symptom nodes involved in the rules, creates bidirectional association edges in the constraint subgraph, and marks the contraindication level and effective time range.

[0110] If there is a conflict between the newly added rules and the node relationship of the existing subgraph, such as inconsistent dosage upper limit rules for the same drug, arbitration is performed based on the rule priority label and effective timestamp, including: high-priority rules override low-priority entries, recently effective rules give priority to inheriting the exception clauses of historical rules, and the conflict record is pushed to the manual review queue.

[0111] When data verification is triggered for the first time after rule injection, the system performs a compatibility stress test on the repaired data, including: randomly selecting batches of historical data to apply the new rules. If the conflict rate exceeds twice the scenario average, the rule is automatically rolled back and marked as "high risk for optimization". At the same time, the temporary execution permission of the old rule is retained until the correction plan is manually confirmed.

[0112] For cross-scenario common rules, such as data privacy protection clauses, the system elevates them to the global constraint layer and inserts reference nodes in all subgraphs to ensure synchronous calls during verification.

[0113] In addition, during the rule expansion process, if it is detected that the connection density of sub-graph nodes is too high, resulting in a decrease in retrieval efficiency, the system automatically starts graph structure optimization, clusters frequently accessed related nodes into independent sub-modules, and establishes a quick index path to accelerate the response speed of multi-level logical verification.

[0114] When the edge node is unable to obtain the newly added rules in real time due to network interruption, the latest valid subgraph copy in the local cache is enabled to perform verification, and the verification results are automatically synchronized to the central knowledge base after the network is restored to complete the missing rule association relationship.

[0115] The data processing system based on artificial intelligence algorithms also includes:

[0116] The load balancing controller is connected to the dynamic metadata perception unit and dynamically allocates data streams to idle nodes to perform feature extraction operations based on the real-time data traffic and CPU occupancy of each edge node.

[0117] It should be further explained that, during the specific implementation process, the load balancing controller continuously monitors the real-time data traffic and CPU occupancy of each edge node. When it detects that the data traffic of a node is close to 80% of the preset peak value or the CPU occupancy exceeds 75%, it initiates a dynamic allocation strategy, including: preferentially redirecting newly connected non-real-time data streams to adjacent idle nodes, while maintaining the processing priority of real-time sensor data streams at the current node to maintain low latency; among them, non-real-time data streams include offline historical logs.

[0118] For data streams that have been assigned to idle nodes, the controller synchronously copies the dynamic feature encoding template of the source node to the target node to ensure the consistency of heterogeneous data parsing rules; if the target node is resource-constrained due to sudden tasks, segmented load migration is enabled to split the single data stream into multiple subtasks and distribute them to at least two nodes for collaborative processing, and verify the consistency of the feature encoding hash values ​​of each subtask before merging the processing results.

[0119] When network delays cause node status information updates to lag behind, the controller predicts resource usage trends within the next 5 seconds based on historical load data. If it predicts that a node is about to be overloaded, it will preheat and migrate some data streams to low-load nodes in advance, and reserve a buffer to prevent data packet loss during the migration process.

[0120] For high-reliability scenarios such as medical care, the controller enables a dual-channel verification mechanism during the migration process, including: the original node retains a copy of the data stream until the target node returns a confirmation signal that the first frame has been successfully processed. If no confirmation is received within the timeout, it triggers a rollback to local processing mode and marks the node as temporarily unavailable.

[0121] In addition, when the overall load of the edge node cluster exceeds the safety threshold, the controller automatically enables the cloud collaboration mode, offloading the non-real-time computing part of the feature extraction task to the cloud, and synchronizes the cloud processing results after the local node completes the real-time feature encoding to supplement the complete metadata chain. At the same time, the cloud collaboration log is recorded for subsequent load strategy optimization analysis; among them, the non-real-time computing part includes semantic conflict backtracking analysis.

[0122] The load balancing controller performs the following operations: when the data flow of the edge node exceeds the preset peak value, it redirects part of the data flow to the adjacent node and maintains the temporal continuity of the dynamic feature encoding during the redirection process.

[0123] It should be further explained that, in the specific implementation process, when the real-time data flow of a certain edge node exceeds the preset peak value, the load balancing controller first identifies the data type attributes of the node, including: for time-sensitive data streams, such as real-time signals from industrial sensors, the neighboring node selection algorithm is immediately triggered to screen out candidate nodes with network delays less than 10 milliseconds and CPU occupancy less than 50%, and the data stream is split into fixed-length data blocks in timestamp order for migration, while retaining copies of the most recent three data blocks at the original node as a rollback buffer; for non-time-series data, such as offline image logs, a hash consistency algorithm is used to disperse the data stream to multiple idle nodes according to content characteristics.

[0124] During the migration process, the controller attaches a metadata tag to each data block, including the source node identifier, timestamp sequence and data block hash value. After receiving it, the target node will first verify the hash value matching. If transmission loss or tampering is detected, the source node will be requested to retransmit the specific data block.

[0125] After the data flow is redirected to the adjacent node, the controller continuously monitors the processing delay of the target node and the output order of the data blocks, including: if the processing delay of the target node causes the overall timing deviation to exceed the business tolerance threshold, such as the millisecond-level deviation of financial transaction data, the virtual timestamp alignment mechanism is automatically inserted to compensate for the delay difference by dynamically adjusting the data block processing interval; if the data block order is detected to be disordered, such as the late-sent data block arriving first due to network jitter, it is re-sorted based on the timestamp in the metadata tag, and an integrity check is performed after the sorting is completed, and the missing data blocks are extracted and supplemented from the rollback buffer.

[0126] In addition, after the migration is completed, if the traffic of the original node falls below the safety threshold, the controller will gradually migrate the new data flow back to the original node and avoid the risk of secondary overload through a progressive weight adjustment strategy, while retaining the adjacent node as a hot standby until the traffic is stable for more than 30 minutes; among them, the progressive weight adjustment strategy includes an initial allocation of 10% of the traffic, followed by a gradient increase of 20%.

[0127] If data flow is interrupted due to node failure during the migration process, the controller activates the emergency channel, forwards the unprocessed data flow to the cloud backup cluster, and performs differential data synchronization after the local edge node is restored to ensure the integrity of the data processing chain.

[0128] A data processing method based on an artificial intelligence algorithm comprises the following steps:

[0129] Step S1: Receive multi-source heterogeneous data streams, including structured databases, unstructured text, and real-time sensor stream data, through the data input interface, and transmit the raw data to the preprocessing engine;

[0130] Step S2: The dynamic metadata perception unit analyzes the data stream in real time at the edge computing node, extracts format features, temporal correlation features, and semantic features, and generates dynamic feature codes. If a data format mutation or source identifier conflict is detected, the feature template is updated and notified to downstream units.

[0131] Step S3: The incremental quality assessment unit calculates integrity, consistency, and timeliness indicators based on dynamic feature coding. When the data source is switched or the format is updated, the assessment threshold is automatically adjusted according to the historical quality gradient changes within the sliding time window, and a quality trend report is generated.

[0132] Step S4: If the data quality does not meet the standards, a repair strategy is generated. After repair, the data enters the rule verification sub-unit, which calls the constraint sub-graph matching the business scenario in the knowledge graph to perform field boundary verification, logical association verification, and cross-source consistency comparison.

[0133] Step S5: After the rule verification passes, the repair data is input into the model to feed back the shadow copy of the verification sub-unit, and the classification confidence fluctuation, feature importance distribution and prediction variance of the downstream model output are monitored. If any indicator exceeds the limit, the repair is deemed to be invalid and the fallback strategy is implemented;

[0134] Step S6: The closed-loop feedback optimization module uses a reinforcement learning strategy to dynamically adjust the repair priority based on the verification results and downstream model performance feedback, prioritizing low latency in real-time scenarios and enabling high-precision multi-round iterative repair in offline scenarios.

[0135] Step S7: The load balancing controller monitors the edge node load in real time. When traffic or resource usage exceeds the limit, it allocates data flows to idle nodes based on data type priority. During the migration process, the time sequence consistency is maintained and data integrity is ensured through hash verification.

[0136] Step S8: When adding a new business scenario or data format, the constraint rules are injected through the knowledge graph interface and stress testing is performed on historical data. If the conflict rate is abnormal, the implementation is delayed and manual review is triggered;

[0137] Step S9: If the edge node resources are insufficient during the processing, the non-real-time tasks are migrated to the cloud for collaborative processing, the core verification function is retained locally, and the difference data is synchronized after the network is restored;

[0138] Step S10: The repair data that finally passes the verification is output to the downstream algorithm model through the preprocessing engine to support intelligent decision-making tasks.

[0139] Through dynamic metadata perception and incremental quality assessment mechanisms, the system reduces the need for manual intervention in the preprocessing stage of multi-source heterogeneous data, and solves the adaptation rigidity problem caused by traditional methods relying on static rules; combined with the two-way verification design of business rules and model feedback, it ensures the dual reliability of data repair strategies in terms of logical rationality and algorithm compatibility, avoiding the risk of overfitting or scenario mismatch caused by a single verification mechanism, thereby improving the input data quality and decision accuracy of downstream artificial intelligence models.

[0140] The system achieves adaptive processing capabilities in complex business scenarios through closed-loop feedback optimization and edge-cloud collaborative load balancing strategies, including: prioritizing low latency and high throughput in real-time streaming data scenarios, and supporting high-precision multi-round repair in offline batch processing; at the same time, the dynamically expanded knowledge graph constraint library and incremental learning mechanism effectively respond to sudden data source switching and new data format access, improving the system's scalability and deployment efficiency.

[0141] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0142] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A data processing system based on artificial intelligence algorithm, characterized in that: include: Data input interface, used to receive multi-source heterogeneous data streams; A preprocessing engine, connected to the data input interface, comprising a dynamic metadata perception unit, an incremental quality assessment unit, a bidirectional verification and self-repair unit, and a closed-loop feedback optimization unit; The dynamic metadata perception unit is deployed on the edge computing node, and extracts the format features, temporal correlation features and semantic features of the data stream in real time through a self-supervised comparative learning framework to generate dynamic feature coding; The incremental quality assessment unit constructs a dynamic weight index including the dimensions of integrity, consistency and timeliness based on the dynamic feature coding, and uses a sliding time window mechanism to update the index threshold in real time; The bidirectional verification self-repair unit includes a rule verification subunit and a model feedback verification subunit, which respectively perform double verification on the repaired data based on the business constraint library of the knowledge graph and the output stability of the downstream model shadow copy; The closed-loop feedback optimization unit generates an optimized repair decision path based on the verification results and downstream model performance feedback through a reinforcement learning strategy; The output end of the preprocessing engine is connected to the algorithm model execution module, which is used to input the verified repair data into the downstream artificial intelligence algorithm.

2. The data processing system based on artificial intelligence algorithm according to claim 1, characterized in that: The dynamic metadata perception unit includes: A lightweight feature extractor uses parameter sharing technology to compress the network size and reduces memory usage through knowledge distillation, so that the feature extractor has low-latency response capabilities at edge nodes.

3. The data processing system based on artificial intelligence algorithm according to claim 2, characterized in that: The incremental quality assessment unit performs the following operations: When a data source switch is detected, the evaluation threshold of the integrity dimension is automatically adjusted based on the gradient change of the historical quality indicator within the sliding time window. The integrity dimension includes the weighted calculation result of the field missing rate and the data coverage rate.

4. The data processing system based on artificial intelligence algorithm according to claim 3, characterized in that: The rule verification subunit includes: The knowledge graph loading module calls the associated constraint subgraph according to the current business scenario type. The constraint subgraph contains causal association rules between data fields, numerical range boundary conditions, and cross-source data consistency rules; The logic verification module performs multi-level matching on the field values ​​of the repaired data and the constraint subgraph, and marks the verification as failed if there is a conflict.

5. The data processing system based on artificial intelligence algorithm according to claim 4, characterized in that: The model feedback verification subunit includes: Shadow copy generation module, which copies the structure and parameters of the downstream algorithm model; The stability monitoring module compares the classification confidence fluctuation, feature importance distribution shift and prediction result variance when the data is input into the shadow copy before and after the repair. If any indicator exceeds the preset tolerance threshold, the repair strategy is triggered to be regenerated.

6. The data processing system based on artificial intelligence algorithm according to claim 5, characterized in that: The closed-loop feedback optimization unit includes: The compound reward calculation module generates reward values ​​based on the verification pass rate of the repair operation, the change rate of the downstream model output standard deviation, and the resource consumption rate of the edge node; The policy optimization module adopts a deep deterministic policy gradient algorithm to dynamically adjust the priority of the repair strategy according to the reward value, so that the repair accuracy and processing delay reach a Pareto optimal balance.

7. The data processing system based on artificial intelligence algorithm according to claim 6, characterized in that: A feature code buffer is provided between the dynamic metadata perception unit and the incremental quality assessment unit for temporarily storing the dynamic feature code in the current time window and sending a trigger signal to the incremental quality assessment unit when the data source is switched.

8. The data processing system based on artificial intelligence algorithm according to claim 7, characterized in that: The knowledge graph loading module further includes: The real-time update interface receives external input constraint rules and automatically expands the node and edge relationships of the constraint subgraph when a new business scenario is added.

9. The data processing system based on artificial intelligence algorithm according to claim 1, characterized in that: The data processing system based on artificial intelligence algorithms also includes: The load balancing controller is connected to the dynamic metadata perception unit and dynamically allocates data streams to idle nodes to perform feature extraction operations based on the real-time data traffic and CPU occupancy of each edge node.

10. The data processing system based on artificial intelligence algorithm according to claim 9, characterized in that: The load balancing controller performs the following operations: when the data flow of the edge node exceeds a preset peak value, redirecting part of the data flow to the adjacent node, and maintaining the temporal continuity of the dynamic feature code during the redirection process.

Images