A data processing system based on artificial intelligence algorithms

Abstract

This invention discloses a data processing system based on artificial intelligence algorithms. The invention relates to the fields of artificial intelligence and data processing technology, and includes: a data input interface for receiving multi-source heterogeneous data streams; and a preprocessing engine connected to the data input interface, comprising a dynamic metadata perception unit, an incremental quality assessment unit, a bidirectional verification self-repair unit, and a closed-loop feedback optimization unit. This data processing system based on artificial intelligence algorithms reduces the need for manual intervention in the preprocessing stage of multi-source heterogeneous data through dynamic metadata perception and incremental quality assessment mechanisms, solving the adaptation rigidity problem caused by traditional methods relying on static rules. The bidirectional verification design, combining business rules and model feedback, ensures the dual reliability of the data repair strategy in terms of logical rationality and algorithm compatibility, avoiding the risk of overfitting or scenario mismatch caused by a single verification mechanism, and improving the input data quality and decision accuracy of downstream artificial intelligence models.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and data processing technology, specifically to a data processing system based on artificial intelligence algorithms. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, its application scenarios in finance, healthcare, and industry are becoming increasingly complex. Data processing, as a fundamental link supporting the efficient operation of algorithm models, directly affects the accuracy and real-time performance of intelligent decision-making. Currently, the data input of AI systems typically involves the integration of multi-source heterogeneous data, such as structured databases, unstructured text, real-time sensor streams, and offline historical records. These data vary significantly in format, scale, update frequency, and quality. Traditional data preprocessing methods rely on manually preset cleaning rules or static quality assessment models, making it difficult to dynamically adapt to frequent changes in business scenarios. Especially in scenarios with high real-time requirements, the dynamic and diverse nature of data streams leads to a significant increase in the cost of manual intervention. Existing systems often require a large amount of time for manual annotation, anomaly detection, and rule adjustment before data fusion, which not only slows down data processing efficiency but may also cause data quality fluctuations due to delayed rule updates, reducing the reliability of input data for subsequent algorithm models. Furthermore, most AI-based data processing systems focus more on optimizing the model itself, such as improving training speed or compressing parameter size, while neglecting the underlying constraints of the data preprocessing stage on algorithm performance. This fragmented design results in a lack of end-to-end automation in the data quality assessment and repair process, preventing the formation of a closed-loop optimization mechanism. When faced with sudden data source switching or the introduction of new data formats, existing systems often cannot quickly identify data characteristics and generate appropriate cleaning strategies, forcing them to rely on manual intervention, which severely restricts the system's scalability and actual deployment efficiency. Summary of the Invention

[0003] (1) Technical problems solved

[0004] To address the shortcomings of existing technologies, this invention provides a data processing system based on artificial intelligence algorithms, which solves the problem of how to achieve automated quality assessment and dynamic repair of multi-source heterogeneous data, thereby reducing reliance on manual intervention and improving the adaptability of data processing processes to complex business scenarios.

[0005] (2) Technical solution

[0006] To achieve the above objectives, the present invention provides the following technical solution: a data processing system based on artificial intelligence algorithms, comprising:

[0007] Data input interface, used to receive multi-source heterogeneous data streams;

[0008] The preprocessing engine, connected to the data input interface, includes a dynamic metadata perception unit, an incremental quality assessment unit, a two-way verification self-healing unit, and a closed-loop feedback optimization unit.

[0009] The dynamic metadata awareness unit is deployed on edge computing nodes and uses a self-supervised contrastive learning framework to extract the format features, temporal correlation features, and semantic features of the data stream in real time to generate dynamic feature codes.

[0010] The incremental quality assessment unit constructs a dynamic weight index that includes dimensions of completeness, consistency and timeliness based on the dynamic feature encoding, and updates the index threshold in real time using a sliding time window mechanism.

[0011] The bidirectional verification self-repair unit includes a rule verification subunit and a model feedback verification subunit, which perform dual verification on the repaired data based on the business constraint library of the knowledge graph and the output stability of the shadow copy of the downstream model, respectively.

[0012] The closed-loop feedback optimization unit generates an optimized repair decision path based on the verification results and downstream model performance feedback through a reinforcement learning strategy.

[0013] The output of the preprocessing engine is connected to the algorithm model execution module, which is used to input the verified repair data into the downstream artificial intelligence algorithm.

[0014] During the operation of the data processing system, after multi-source heterogeneous data is input into the system through the data input interface, it first enters the metadata perception stage of the preprocessing engine.

[0015] The dynamic metadata awareness unit uses a lightweight feature extractor to perform real-time parsing and feature capture of the input data stream.

[0016] For structured data, the extractor identifies field names, data types, and numerical distribution patterns, and automatically detects missing fields or mismatched types. For unstructured text, it uses semantic context analysis to locate keyword conflicts and logical breakpoints. For real-time sensor data streams, the extractor synchronously tracks timestamp sequences and numerical change trends to identify abnormal jitter or continuity interruptions.

[0017] When the data stream format changes abruptly, the extractor performs a difference comparison based on the historical feature encoding template. If a new field or format version upgrade is detected, a feature encoding template update notification is immediately triggered, and a format change identifier is sent to the incremental quality assessment unit.

[0018] Preferably, the dynamic metadata sensing unit includes:

[0019] The lightweight feature extractor uses parameter sharing technology to compress the network size and reduces memory usage through knowledge distillation, enabling the feature extractor to have low-latency response capabilities at edge nodes.

[0020] Preferably, the incremental quality assessment unit performs the following operations:

[0021] When a data source switch is detected, the evaluation threshold of the integrity dimension is automatically adjusted based on the gradient changes of historical quality indicators within a sliding time window. The integrity dimension includes the weighted calculation result of field missing rate and data coverage.

[0022] After receiving dynamic feature codes, the incremental quality assessment unit initiates multi-dimensional quality indicator calculations. The integrity dimension dynamically sets thresholds based on the historical distribution of field missing rates and data coverage. This includes: when a data source switch causes a sudden increase in field missing rates but the data coverage does not decrease synchronously, it is determined to be a format compatibility issue, the integrity threshold is temporarily relaxed, and cross-source consistency enhancement checks are initiated. The consistency dimension identifies naming conflicts or unit format mismatches by comparing the field mapping relationship between the current data stream and associated data sources. The timeliness dimension dynamically adjusts the timeout tolerance window for real-time streaming data based on the difference between the data generation timestamp and the processing delay.

[0023] When the quality index fluctuation exceeds the preset sensitivity threshold within the sliding time window, the evaluation unit automatically generates a quality trend report and triggers the threshold adaptive adjustment mechanism, including: if the fluctuation is caused by an occasional anomaly, the threshold curve is smoothed using a moving average algorithm; if the fluctuation shows a continuous upward trend, the root cause analysis module is activated to trace the data source or parser anomaly.

[0024] Preferably, the rule verification subunit includes:

[0025] The knowledge graph loading module calls the associated constraint subgraph according to the current business scenario type. The constraint subgraph contains causal relationship rules between data fields, numerical range boundary conditions, and cross-source data consistency rules.

[0026] The logic verification module performs multi-level matching between the field values ​​of the repaired data and the constraint subgraph. If a conflict exists, it is marked as a verification failure.

[0027] When the repaired data enters the two-way verification phase, the rule verification subunit calls the constraint subgraph in the knowledge graph that matches the current business scenario. For example, in a financial transaction scenario, a subgraph containing rules for account transaction continuity, reasonable boundaries for transaction amounts, and anti-money laundering related conditions is loaded to perform hierarchical verification on the repaired data. This includes: first, verifying whether the amount of a single transaction exceeds the account's historical transaction range; second, verifying whether the time sequence of multiple transactions from the same account is logical; and finally, cross-referencing with the blacklist of an external credit reporting system.

[0028] If the verification finds that the transaction amount is abnormal but the time sequence is reasonable, dynamic threshold correction is triggered, and the reasonable range is recalculated based on the average transaction value of the account over the past three months; if the time sequence is conflicting and the associated account has a blacklist record, the batch of data is frozen and a high-risk alert is generated.

[0029] Preferably, the model feedback verification subunit includes:

[0030] The shadow copy generation module replicates the structure and parameters of the downstream algorithm model.

[0031] The stability monitoring module compares the fluctuations in classification confidence, feature importance distribution shifts, and prediction result variances when data is input into the shadow copy before and after repair. If any of these indicators exceeds a preset tolerance threshold, the repair strategy is regenerated. The model feedback verification subunit monitors the stability changes of the model output after inputting the repaired data into the shadow copy. This includes: for image recognition tasks, statistically analyzing the similarity of feature vectors and the difference in classification results before and after repair; for time-series prediction tasks, analyzing the smoothness of the prediction curve and its deviation from the historical baseline. If the similarity decreases beyond the tolerance or the prediction deviation continues to widen, the repair strategy is deemed ineffective, automatically reverting to the previous effective version and marking the abnormal data source.

[0032] Preferably, the closed-loop feedback optimization unit includes:

[0033] The composite reward calculation module generates reward values ​​based on the verification pass rate of repair operations, the rate of change of the standard deviation of downstream model output, and the resource consumption rate of edge nodes.

[0034] The strategy optimization module employs a deep deterministic strategy gradient algorithm to dynamically adjust the priority of the repair strategy based on the reward value, thereby achieving a Pareto optimal balance between repair accuracy and processing latency.

[0035] The closed-loop feedback optimization module dynamically optimizes the repair strategy based on the verification results and downstream model performance feedback. In real-time stream processing scenarios, the system prioritizes low latency requirements and adopts a fast repair strategy to skip deep validation of non-critical fields; in offline batch processing scenarios, a high-precision mode is enabled, allowing cross-batch data backtracking and multiple rounds of iterative repair.

[0036] When resource constraints lead to insufficient computing power at edge nodes, the system automatically migrates non-real-time tasks to the cloud and retains the minimum necessary verification functions locally to ensure continuous processing of core data streams.

[0037] Preferably, a feature encoding cache is provided between the dynamic metadata perception unit and the incremental quality assessment unit to temporarily store the dynamic feature encoding within the current time window and send a trigger signal to the incremental quality assessment unit when the data source is switched.

[0038] Preferably, the knowledge graph loading module further includes:

[0039] The interface is updated in real time. When a new business scenario is added, the constraint rules input from the outside are received and the node and edge relationships of the constraint subgraph are automatically expanded.

[0040] When new business scenarios or data formats are added, the system injects new rules through the knowledge graph update interface and performs stress tests during the first verification, including: extracting historical data to apply new rules; if the conflict rate increases abnormally, the rules are delayed and manual review is triggered; if the test passes, the knowledge graph node connection path is automatically optimized to improve the efficiency of subsequent verification.

[0041] For tasks migrated to the cloud, the system records the complete metadata chain and processing logs to ensure data traceability and consistency between local and cloud environments, and automatically synchronizes the difference data after the network is restored.

[0042] Preferably, the data processing system based on artificial intelligence algorithms also includes:

[0043] The load balancer controller, connected to the dynamic metadata sensing unit, dynamically allocates data streams to idle nodes for feature extraction operations based on the real-time data traffic and CPU utilization of each edge node. The load balancer controller monitors the node load status in real time. When the processing latency of a node approaches a threshold, it dynamically allocates data streams to idle nodes and maintains the temporal continuity and integrity verification of data blocks during the migration process to prevent data out-of-ordering or loss due to network jitter.

[0044] Preferably, the load balancer performs the following operations: when the data traffic of the edge node exceeds a preset peak, it redirects part of the data traffic to the adjacent node, and maintains the temporal continuity of the dynamic feature encoding during the redirection process.

[0045] Through the implementation of the above solution, the system achieves fully automated processing of multi-source heterogeneous data from access, quality assessment, repair verification to closed-loop optimization, reducing the need for manual intervention while ensuring the accuracy and real-time performance of data processing in complex business scenarios. The collaborative mechanism and adaptive strategy design between modules effectively solve the problems of rigid rules, single verification, and insufficient scalability in traditional systems when performing dynamic data fusion.

[0046] (3) Beneficial effects

[0047] This invention provides a data processing system based on artificial intelligence algorithms. It has the following beneficial effects:

[0048] (i) This data processing system based on artificial intelligence algorithms reduces the need for manual intervention in the preprocessing stage of multi-source heterogeneous data through dynamic metadata perception and incremental quality assessment mechanisms, and solves the adaptation rigidity problem caused by the reliance on static rules in traditional methods. Combined with the two-way verification design of business rules and model feedback, it ensures the dual reliability of data repair strategies in terms of logical rationality and algorithm compatibility, avoids the risk of overfitting or scenario mismatch caused by a single verification mechanism, and thus improves the input data quality and decision accuracy of downstream artificial intelligence models.

[0049] (II) This data processing system based on artificial intelligence algorithms achieves adaptive processing capabilities in complex business scenarios through closed-loop feedback optimization and edge-cloud collaborative load balancing strategies. These capabilities include: prioritizing low latency and high throughput in real-time streaming data scenarios, and supporting high-precision multi-round repair in offline batch processing. Meanwhile, the dynamically expanded knowledge graph constraint library and incremental learning mechanism effectively cope with sudden data source switching and the access of new data formats, thereby improving the system's scalability and deployment efficiency. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0051] Figure 2 This is the timing diagram of the control logic of the present invention. Detailed Implementation

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] Please see Figure 1 and Figure 2 This invention provides a technical solution: a data processing system based on artificial intelligence algorithms, comprising:

[0054] Data input interface, used to receive multi-source heterogeneous data streams;

[0055] The preprocessing engine, connected to the data input interface, includes a dynamic metadata awareness unit, an incremental quality assessment unit, a two-way verification self-healing unit, and a closed-loop feedback optimization unit.

[0056] Among them, the dynamic metadata awareness unit is deployed on edge computing nodes. It extracts the format features, temporal correlation features and semantic features of the data stream in real time through a self-supervised contrastive learning framework to generate dynamic feature codes.

[0057] The incremental quality assessment unit is based on dynamic feature encoding, constructs dynamic weight indicators that include dimensions of completeness, consistency and timeliness, and uses a sliding time window mechanism to update the indicator thresholds in real time.

[0058] The bidirectional verification self-repair unit includes a rule verification subunit and a model feedback verification subunit, which perform dual verification on the repaired data based on the business constraint library of the knowledge graph and the output stability of the shadow copy of the downstream model, respectively.

[0059] The closed-loop feedback optimization unit generates an optimized repair decision path based on the verification results and downstream model performance feedback through a reinforcement learning strategy.

[0060] The output of the preprocessing engine is connected to the algorithm model execution module, which is used to input the verified repair data into the downstream artificial intelligence algorithm.

[0061] The dynamic metadata awareness unit includes:

[0062] The lightweight feature extractor uses parameter sharing technology to compress the network size and reduces memory usage through knowledge distillation, resulting in a response latency of less than 5 milliseconds at edge nodes.

[0063] It should be further explained that, in the specific implementation process, when the lightweight feature extractor runs at the edge node, it first merges the redundant convolutional kernels and fully connected layer weights in the multi-layer neural network through parameter sharing technology to reduce the number of network parameters; at the same time, it adopts a knowledge distillation framework to transfer the feature mapping relationship in the pre-trained large metadata perception model to the lightweight model, which retains the key feature recognition capability while reducing memory usage.

[0064] When the data stream enters the edge node, the extractor parses the format and metadata of the data packets in real time. It automatically identifies missing fields or mismatched types for structured data, detects semantic conflicts and context breaks for unstructured text, and triggers a dynamic sampling rate adjustment mechanism to balance processing accuracy and latency when it detects anomalies in the temporal correlation of real-time streaming data.

[0065] In edge computing resource-constrained scenarios, if the memory occupancy rate of the current node exceeds a preset threshold, the extractor automatically switches to a low-power mode, retaining only the core feature extraction layer to run, and temporarily caching the segmented hash values ​​of unprocessed data streams. Once resources are released, feature encoding generation of high-priority data is prioritized.

[0066] In addition, when a sudden increase in data flow rate causes the response latency to approach the 5-millisecond limit, the extractor dynamically allocates computing threads to idle cores based on historical load data collected by sliding time windows, and skips secondary verification of non-critical features to ensure that sensor data with the highest real-time requirements is processed first.

[0067] The incremental quality assessment unit performs the following operations:

[0068] When a data source switch is detected, the evaluation threshold of the integrity dimension is automatically adjusted based on the gradient changes of historical quality indicators within a sliding time window. The integrity dimension includes a weighted calculation result of field missing rate and data coverage.

[0069] It should be further explained that, in the specific implementation process, when the system detects a data source switch or a data format update, the incremental quality assessment unit first obtains the changing trend of historical quality indicators within the current sliding time window. By calculating the gradient difference of the weighted values ​​of field missing rate and data coverage in the integrity dimension over continuous time segments, it determines whether the data quality fluctuation is caused by the source switch.

[0070] If the gradient difference exceeds the preset sensitivity threshold, a dynamic evaluation threshold adjustment mechanism will be activated, including: for data streams from structured databases, the tolerance upper limit for field missing rates will be reset based on the field naming conventions and historical coverage distribution of the new data source; for unstructured text data streams, the evaluation weight of the timeliness dimension will be dynamically relaxed based on the semantic coherence detection results to avoid misjudgments caused by format parsing delays.

[0071] When the integrity index fluctuates steadily over three consecutive time windows after the data source switch, the original evaluation threshold is gradually restored, and the adjusted parameters are recorded in the historical knowledge base for quick access in subsequent similar scenarios.

[0072] If a data format update causes a sudden increase in the field missing rate but the data coverage does not decrease synchronously, it is determined to be a format compatibility conflict, triggering cross-source consistency rule strengthening verification, and temporarily freezing the evaluation weight update of the timeliness dimension until the format parser completes adaptive calibration.

[0073] In addition, when historical data is insufficient or gradient changes are abnormally weak, a hybrid strategy combining the default threshold and the moving average prediction value is adopted to prevent threshold drift caused by data sparsity, and a source switching quality assessment report is generated on the interface for operation and maintenance personnel to review.

[0074] The rule validation subunit includes:

[0075] The knowledge graph loading module calls the associated constraint subgraphs based on the current business scenario type. The constraint subgraphs contain causal relationship rules between data fields, numerical range boundary conditions, and cross-source data consistency rules.

[0076] The logic verification module performs multi-level matching between the field values ​​of the repaired data and the constraint subgraph. If a conflict exists, it is marked as a verification failure.

[0077] It should be further explained that, in the specific implementation process, when the repaired data enters the rule verification sub-unit, the system loads the corresponding constraint sub-graph from the knowledge graph according to the current business scenario type. For example, in a medical scenario, the sub-graph containing the association rules between patient age and diagnosis results, drug dosage range boundaries, and cross-departmental data consistency conditions is invoked to perform multi-level verification on the field values ​​of the repaired data. This includes: first, checking whether a single field violates numerical boundaries, such as blood pressure values ​​exceeding physiological limits; second, verifying cross-field logical relationships, such as the temporal sequence matching of surgical records and anesthesia dosages; and finally, cross-comparing with external data sources, such as the compliant mapping between medical insurance codes and treatment items.

[0078] If a certain level of verification fails, differentiated processing is performed based on the type of conflict, including: for numerical out-of-bounds errors, an automatic threshold dynamic correction mechanism is triggered, and temporary boundaries are redefined based on historical data distribution; for logical association conflicts, the data repair path is traced and the fields causing the contradiction are isolated, and manual review marks are generated; for cross-source consistency inconsistencies, the data batch is frozen first and cross-system data source synchronization verification is initiated.

[0079] When a new business scenario is added, after injecting new constraint rules through the real-time update interface, the system automatically splits the general rules and scenario-specific rules in the original subgraph, and performs compatibility testing on the new rules during the first verification, including: if the conflict rate between the repaired data and the new rules exceeds a preset threshold, the rule effective time is delayed and a rule conflict analysis report is triggered until the rule priority ranking is manually confirmed.

[0080] In addition, if a complex conflict involving multiple related fields is detected within the same data batch during the verification process, the system will calculate the conflict weight based on the node connection density of the knowledge graph and prioritize resolving high-weight conflicts to reduce the risk of cascading errors.

[0081] The model feedback validation subunit includes:

[0082] The shadow copy generation module replicates the structure and parameters of the downstream algorithm model.

[0083] The stability monitoring module compares the fluctuations in classification confidence, feature importance distribution shifts, and prediction result variances when inputting shadow copies of data before and after the repair. If any of these indicators exceeds a preset tolerance threshold, the repair strategy will be regenerated.

[0084] It should be further explained that, in the specific implementation process, when the data input model is repaired and fed back to the verification subunit, the system first copies the structure and parameters of the downstream algorithm model to generate a shadow copy, ensuring that its initial state is completely consistent with the production environment model.

[0085] Subsequently, the data before and after the repair are input into the shadow copy to monitor the fluctuation of the classification confidence of the model output. This includes: if the difference between the highest class confidence of the same data sample before and after the repair exceeds the preset tolerance threshold, it is judged as a potential repair distortion; at the same time, the feature importance distribution shift is analyzed, and the abnormal shift of model attention caused by data repair is identified by calculating the standard deviation of the change in the weight of key features.

[0086] For time series prediction tasks, the variance change rate of the prediction results is additionally calculated. If the predicted variance of the corrected data exceeds the historical baseline range three times in a row, an early warning mechanism is triggered.

[0087] When any of the above indicators exceed the limit, the system will automatically trace the repair operation steps that caused the anomaly. If the step involves field type conversion or missing value filling, the current repair strategy will be frozen first and rolled back to the previous effective version. If the anomaly is caused by a parsing error of the newly added data source, the data source will be temporarily blocked and the cross-source verification process will be started.

[0088] In financial risk control scenarios, if classification confidence fluctuations and feature importance shifts occur simultaneously but do not exceed a threshold, the system will correlate the trends of the two changes. This includes: when the direction of confidence fluctuations is negatively correlated with the direction of feature weight migration, it is determined that the model robustness has decreased, and the priority of the current repair strategy needs to be reduced and the verification time extended; if they are positively correlated and the prediction variance is stable, the repair is considered effective, but additional manual sampling verification is still required.

[0089] In addition, when some indicators in the same batch of data exceed the limit while others are normal, the system performs differential processing based on the business risk level. This includes: for highly sensitive businesses, such as medical diagnosis, immediately interrupting the process and generating an anomaly report; for low-risk businesses, such as product recommendation, allowing the data to enter the production model but marking it as having a downgraded confidence level, and strengthening the verification weight of this scenario in subsequent feedback optimization.

[0090] The closed-loop feedback optimization unit includes:

[0091] The composite reward calculation module generates reward values ​​based on the verification pass rate of repair operations, the rate of change of the standard deviation of downstream model output, and the resource consumption rate of edge nodes.

[0092] The strategy optimization module employs a deep deterministic strategy gradient algorithm to dynamically adjust the priority of the repair strategy based on the reward value, thereby achieving a Pareto optimal balance between repair accuracy and processing latency.

[0093] It should be further explained that, in the specific implementation process, when the composite reward calculation module generates the reward value, it first dynamically allocates the weight coefficients of each indicator according to the preset business scenario type, including: increasing the weight ratio of processing latency in real-time sensitive scenarios, and strengthening the impact factor of verification pass rate in accuracy sensitive scenarios.

[0094] For the verification pass rate of the repair operation, the percentage of data batches that pass the double verification within the current time window is statistically analyzed, and the relative improvement is calculated in combination with the historical average pass rate; the performance improvement rate of the downstream model is determined by comparing the output stability of the same batch of data input into the production model before and after the repair. If the standard deviation of the highest confidence in the classification task decreases or the fluctuation range of the prediction error in the regression task narrows, it is judged as a positive improvement.

[0095] The resource consumption rate is calculated by combining the CPU utilization, peak memory usage, and network transmission latency of edge nodes. When the resource consumption of a node exceeds twice the average level of the cluster, a penalty coefficient is applied to the repair operation initiated by that node.

[0096] When the strategy optimization module adopts the deep deterministic strategy gradient algorithm, it dynamically adjusts the balance between exploration and utilization of the repair strategy based on the real-time reward value. This includes: if the reward value shows an upward trend within three consecutive time windows, the proportion of random exploration is reduced to stabilize the strategy; if the reward value fluctuates beyond the historical variance threshold, exploration actions are increased to escape local optima.

[0097] When processing high-concurrency data streams, priority queues are set for different data types, including: real-time sensor streams adopt a low-latency repair strategy, prioritizing skipping non-critical field validation to maintain throughput; offline historical data is enabled in a high-precision repair mode, allowing cross-batch data backtracking to complete missing fields.

[0098] If the system detects that the repair strategy for a certain type of data source has been in a low-reward state for a long time, it will trigger the strategy rollback mechanism, automatically switch to the historical best strategy version, and mark the data source with a characteristic code for manual review.

[0099] In addition, in resource-constrained edge nodes, when the composite reward value continues to decline due to insufficient computing resources, the strategy optimization module will force the degradation of some repair functions to the cloud for collaborative processing, while retaining the minimum necessary verification logic locally to maintain the basic service quality.

[0100] A feature encoding cache is set up between the dynamic metadata perception unit and the incremental quality assessment unit to temporarily store the dynamic feature encoding within the current time window and send a trigger signal to the incremental quality assessment unit when the data source is switched.

[0101] It should be further explained that, in the specific implementation process, when the feature code generated by the dynamic metadata perception unit is transmitted to the feature code cache, the system stores the code in segments according to the preset time window length, and records the generation timestamp and data source identifier of each segment.

[0102] If a change in the data source identifier is detected within an adjacent time window, it is determined to be a normal data source switching event. The cache immediately compresses all the encodings in the current window into a snapshot file, attaches a source switching mark, and sends it to the incremental quality assessment unit. If a mixed data source identifier appears in the same time window without a clear switching signal, a sudden switching detection mechanism is triggered. The difference in format features in the encoding is extracted first for conflict analysis, including: when the coexistence ratio of structured and unstructured data encodings exceeds the threshold, the cache is automatically divided into multi-source independent storage blocks, and conflict marks are attached to each block for subsequent evaluation unit differential processing.

[0103] When edge node storage resources are insufficient, the cache initiates a dynamic cleanup strategy, including: for encoded segments that have been sent to the evaluation unit and passed the initial verification, only the hash digest is retained and the original data space is released; for encoded segments that have not been sent, low-priority content is selectively discarded according to the data priority label, and the metadata of the lost encoding is recorded in the log for traceability.

[0104] When the data source switching trigger signal is sent, if the incremental quality assessment unit reports that the assessment threshold adjustment failed, the cache will reload the encoding snapshot of the previous time window and compare it with the current encoding. If the key feature matching degree is lower than the preset value, it is determined that the data source metadata is mislabeled, and the perception unit is triggered to re-extract the feature encoding.

[0105] In addition, in high-concurrency scenarios, if the buffer receiving encoding rate continuously exceeds the processing capacity of the evaluation unit, a traffic shaping mechanism is activated, including: enabling a pass-through channel for real-time sensor data encoding to bypass the buffer and directly transmit it to the evaluation unit for streaming processing, while reducing the sampling frequency of offline data encoding to balance the load.

[0106] The knowledge graph loading module further includes:

[0107] The interface is updated in real time. When a new business scenario is added, it receives externally input constraint rules and automatically expands the node and edge relationships of the constraint subgraph.

[0108] It should be further explained that, in the specific implementation process, when a new business scenario is added, the system receives the constraint rules input from the outside through the real-time update interface. First, it performs a syntax compliance check, including: if the rule contains undefined data fields or logical operator conflicts, the injection is rejected and an error report is generated; the rules that pass the verification are broken down into atomic condition units and automatically mapped to existing nodes in the knowledge graph according to the semantic relevance.

[0109] For newly added drug incompatibilities rules in medical scenarios, the system identifies the drug component nodes and symptom nodes involved in the rules, creates bidirectional association edges in the constraint subgraph, and marks the incompatibility level and effective time range.

[0110] If a new rule conflicts with the node relationships of an existing subgraph, such as the dosage limit rule for the same drug being inconsistent, arbitration will be conducted based on the rule priority label and effective timestamp. This includes: higher priority rules covering lower priority entries, and recently effective rules inheriting exception clauses from historical rules. The conflict record will then be pushed to the manual review queue.

[0111] When data verification is triggered for the first time after rule injection, the system performs a compatibility stress test on the repaired data, including: randomly selecting historical data batches to apply the new rule; if the conflict rate exceeds twice the average of the scenario, the rule is automatically rolled back and marked as "high risk and to be optimized", while the temporary execution permission of the old rule is retained until the correction plan is manually confirmed.

[0112] For rules that are common across scenarios, such as data privacy protection clauses, the system elevates them to the global constraint layer and inserts reference nodes in all subgraphs to ensure synchronous invocation during verification.

[0113] In addition, if the system detects that the subgraph node connection density is too high during the rule expansion process, resulting in a decrease in retrieval efficiency, the system will automatically start graph structure optimization, cluster frequently accessed related nodes into independent sub-modules, and establish a shortcut index path to accelerate the response speed of multi-level logical verification.

[0114] When edge nodes are unable to obtain new rules in real time due to network interruption, the latest valid subgraph copy cached locally is used to perform verification, and the verification results are automatically synchronized to the central knowledge base after the network is restored to fill in the missing rule relationships.

[0115] Data processing systems based on artificial intelligence algorithms also include:

[0116] The load balancer controller, connected to the dynamic metadata sensing unit, dynamically allocates data streams to idle nodes to perform feature extraction operations based on the real-time data traffic and CPU utilization of each edge node.

[0117] It should be further explained that, in the specific implementation process, the load balancer continuously monitors the real-time data traffic and CPU utilization of each edge node. When it detects that the data traffic of a certain node is close to 80% of the preset peak or the CPU utilization exceeds 75%, a dynamic allocation strategy is launched, including: prioritizing the redirection of newly connected non-real-time data streams to adjacent idle nodes, while maintaining the processing priority of real-time sensor data streams on the current node to maintain low latency; among them, non-real-time data streams include offline historical logs.

[0118] For data streams already allocated to idle nodes, the controller synchronously copies the dynamic feature encoding template of the source node to the target node to ensure the consistency of heterogeneous data parsing rules. If the target node experiences resource shortages due to sudden tasks, segmented load migration is enabled, splitting a single data stream into multiple sub-tasks and distributing them to at least two nodes for collaborative processing. Before merging the processing results, the consistency of the feature encoding hash values ​​of each sub-task is verified.

[0119] When network latency causes a delay in updating node status information, the controller predicts the resource usage trend within the next 5 seconds based on historical load data. If it predicts that a node is about to become overloaded, it will pre-warm and migrate some data streams to low-load nodes and reserve a buffer to prevent data packet loss during the migration process.

[0120] For high-reliability scenarios such as medical care, the controller enables a dual-channel verification mechanism during the migration process, including: the original node retains a copy of the data stream until the target node returns an acknowledgment signal that the first frame was successfully processed; if no acknowledgment is received within the timeout period, a rollback to the local processing mode is triggered and the node is marked as temporarily unavailable.

[0121] In addition, when the overall load of the edge node cluster exceeds the safety threshold, the controller automatically enables the cloud collaboration mode, offloading the non-real-time computing part of the feature extraction task to the cloud, and synchronizing the cloud processing results after the local node completes the real-time feature encoding to supplement the complete metadata chain. At the same time, cloud collaboration logs are recorded for subsequent load strategy optimization analysis. The non-real-time computing part includes semantic conflict backtracking analysis.

[0122] The load balancer controller performs the following operations: when the data traffic of the edge node exceeds the preset peak, it redirects part of the data flow to the adjacent node, and maintains the temporal continuity of the dynamic feature encoding during the redirection process.

[0123] It should be further explained that, in the specific implementation process, when the real-time data traffic of a certain edge node exceeds the preset peak, the load balancer first identifies the data type attributes of the node, including: for time-sensitive data streams, such as real-time signals from industrial sensors, the nearest node selection algorithm is immediately triggered to filter out candidate nodes with network latency of less than 10 milliseconds and CPU utilization of less than 50%, and the data stream is split into fixed-length data blocks according to timestamp order for migration, while retaining copies of the three most recent data blocks on the original node as a rollback buffer; for non-time-series data, such as offline image logs, the hash consistency algorithm is used to distribute the data stream to multiple idle nodes according to content characteristics.

[0124] During the migration process, the controller attaches metadata tags to each data block, including the source node identifier, timestamp sequence, and data block hash value. After receiving the data, the target node first verifies the hash value matching. If packet loss or tampering is detected, it requests the source node to retransmit the specific data block.

[0125] After the data stream is redirected to an adjacent node, the controller continuously monitors the processing latency of the target node and the output order of data blocks. This includes: if the processing latency of the target node causes the overall timing deviation to exceed the business tolerance threshold, such as a millisecond-level deviation in financial transaction data, a virtual timestamp alignment mechanism is automatically inserted to compensate for the latency difference by dynamically adjusting the data block processing interval; if a data block order disorder is detected, such as a later data block arriving first due to network jitter, the data blocks are reordered based on the timestamp in the metadata tag, and an integrity check is performed after the sorting is completed. Missing data blocks are retrieved from the rollback buffer to make up for them.

[0126] In addition, if the traffic of the original node falls below the safety threshold after the migration is completed, the controller will gradually migrate the new data stream back to the original node and avoid the risk of secondary overload through a progressive weight adjustment strategy. At the same time, it will retain adjacent nodes as hot backups until the traffic stabilizes for more than 30 minutes. The progressive weight adjustment strategy includes an initial allocation of 10% of the traffic, followed by a 20% increment.

[0127] In the event of data flow interruption due to node failure during the migration process, the controller activates an emergency channel to forward unprocessed data flow to the cloud backup cluster and performs differential data synchronization after the local edge node recovers, ensuring the integrity of the data processing chain.

[0128] A data processing method based on artificial intelligence algorithms includes the following steps:

[0129] Step S1: Receive multi-source heterogeneous data streams through the data input interface, including structured databases, unstructured text and real-time sensor stream data, and transmit the raw data to the preprocessing engine;

[0130] Step S2: The dynamic metadata awareness unit parses the data stream in real time at the edge computing node, extracts format features, temporal correlation features and semantic features, and generates dynamic feature codes; if a data format mutation or source identifier conflict is detected, the feature template is updated and the downstream unit is notified.

[0131] Step S3: The incremental quality assessment unit calculates integrity, consistency and timeliness indicators based on dynamic feature coding. When the data source is switched or the format is updated, the assessment threshold is automatically adjusted according to the historical quality gradient changes within the sliding time window, and a quality trend report is generated.

[0132] Step S4: If the data quality does not meet the standard, a repair strategy is generated. After repair, the data enters the rule verification sub-unit, which calls the constraint sub-graph in the knowledge graph that matches the business scenario, and performs field boundary verification, logical association verification, and cross-source consistency comparison.

[0133] Step S5: After the rule verification is passed, the repaired data will be input into the model to feed back the shadow copy of the verification subunit. The fluctuation of classification confidence, feature importance distribution and prediction variance of the downstream model output will be monitored. If any indicator exceeds the limit, the repair will be deemed to have failed and the strategy will be rolled back.

[0134] Step S6: The closed-loop feedback optimization module dynamically adjusts the repair priority based on the verification results and downstream model performance feedback, using a reinforcement learning strategy. In real-time scenarios, it prioritizes low latency, while in offline scenarios, it enables high-precision multi-round iterative repair.

[0135] Step S7: The load balancer monitors the load of edge nodes in real time. When traffic or resource usage exceeds the limit, it allocates data streams to idle nodes according to data type priority. During the migration process, it maintains the continuity of time sequence and ensures data integrity through hash verification.

[0136] Step S8: When adding new business scenarios or data formats, inject constraint rules through the knowledge graph interface and perform stress tests on historical data. If the conflict rate is abnormal, the effect will be delayed and manual review will be triggered.

[0137] Step S9: If edge node resources are insufficient during processing, non-real-time tasks will be migrated to the cloud for collaborative processing, while the core verification function will be retained locally, and the difference data will be synchronized after the network is restored.

[0138] Step S10: The final verified repair data is output to the downstream algorithm model through the preprocessing engine to support intelligent decision-making tasks.

[0139] The system reduces the need for manual intervention in the preprocessing stage of multi-source heterogeneous data through dynamic metadata awareness and incremental quality assessment mechanisms, and solves the adaptation rigidity problem caused by the reliance on static rules in traditional methods. Combined with the two-way verification design of business rules and model feedback, it ensures the dual reliability of data repair strategies in terms of logical rationality and algorithm compatibility, avoids the risk of overfitting or scenario mismatch caused by a single verification mechanism, and thus improves the input data quality and decision accuracy of downstream artificial intelligence models.

[0140] The system achieves adaptive processing capabilities in complex business scenarios through closed-loop feedback optimization and edge-cloud collaborative load balancing strategies. This includes prioritizing low latency and high throughput in real-time streaming data scenarios, and supporting high-precision multi-round repair in offline batch processing. At the same time, the dynamically expanded knowledge graph constraint library and incremental learning mechanism effectively cope with sudden data source switching and the access of new data formats, improving the system's scalability and deployment efficiency.

[0141] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0142] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A data processing system based on artificial intelligence algorithms, characterized in that, include: Data input interface, used to receive multi-source heterogeneous data streams; The preprocessing engine, connected to the data input interface, includes a dynamic metadata perception unit, an incremental quality assessment unit, a two-way verification self-healing unit, and a closed-loop feedback optimization unit. The dynamic metadata awareness unit is deployed on edge computing nodes and uses a self-supervised contrastive learning framework to extract the format features, temporal correlation features, and semantic features of the data stream in real time to generate dynamic feature codes. The incremental quality assessment unit constructs a dynamic weight index that includes dimensions of completeness, consistency and timeliness based on the dynamic feature encoding, and updates the index threshold in real time using a sliding time window mechanism. The bidirectional verification self-repair unit includes a rule verification subunit and a model feedback verification subunit, which perform dual verification on the repaired data based on the business constraint library of the knowledge graph and the output stability of the shadow copy of the downstream model, respectively. The closed-loop feedback optimization unit generates an optimized repair decision path based on the verification results and downstream model performance feedback through a reinforcement learning strategy. The output of the preprocessing engine is connected to the algorithm model execution module, which is used to input the verified repair data into the downstream artificial intelligence algorithm.

2. The data processing system based on artificial intelligence algorithms according to claim 1, characterized in that: The dynamic metadata sensing unit includes: The lightweight feature extractor uses parameter sharing technology to compress the network size and reduces memory usage through knowledge distillation, enabling the feature extractor to have low-latency response capabilities at edge nodes.

3. The data processing system based on artificial intelligence algorithms according to claim 2, characterized in that: The incremental quality assessment unit performs the following operations: When a data source switch is detected, the evaluation threshold of the integrity dimension is automatically adjusted based on the gradient changes of historical quality indicators within a sliding time window. The integrity dimension includes the weighted calculation result of field missing rate and data coverage.

4. A data processing system based on artificial intelligence algorithms according to claim 3, characterized in that: The rule verification subunit includes: The knowledge graph loading module calls the associated constraint subgraph according to the current business scenario type. The constraint subgraph contains causal relationship rules between data fields, numerical range boundary conditions, and cross-source data consistency rules. The logic verification module performs multi-level matching between the field values ​​of the repaired data and the constraint subgraph. If a conflict exists, it is marked as a verification failure.

5. A data processing system based on artificial intelligence algorithms according to claim 4, characterized in that: The model feedback verification subunit includes: The shadow copy generation module replicates the structure and parameters of the downstream algorithm model. The stability monitoring module compares the fluctuation of classification confidence, the shift in feature importance distribution, and the variance of prediction results when the data is input into the shadow copy before and after the repair. If any of the indicators exceeds the preset tolerance threshold, the repair strategy is triggered to be regenerated.

6. A data processing system based on artificial intelligence algorithms according to claim 5, characterized in that: The closed-loop feedback optimization unit includes: The composite reward calculation module generates reward values ​​based on the verification pass rate of repair operations, the rate of change of the standard deviation of downstream model output, and the resource consumption rate of edge nodes. The strategy optimization module employs a deep deterministic strategy gradient algorithm to dynamically adjust the priority of the repair strategy based on the reward value, thereby achieving a Pareto optimal balance between repair accuracy and processing latency.

7. A data processing system based on artificial intelligence algorithms according to claim 6, characterized in that: A feature encoding cache is provided between the dynamic metadata perception unit and the incremental quality assessment unit to temporarily store the dynamic feature encoding within the current time window and send a trigger signal to the incremental quality assessment unit when the data source is switched.

8. A data processing system based on an artificial intelligence algorithm according to claim 7, characterized in that: The knowledge graph loading module further includes: The interface is updated in real time. When a new business scenario is added, the constraint rules input from the outside are received and the node and edge relationships of the constraint subgraph are automatically expanded.

9. A data processing system based on artificial intelligence algorithms according to claim 1, characterized in that: Data processing systems based on artificial intelligence algorithms also include: The load balancer controller is connected to the dynamic metadata sensing unit and dynamically allocates data streams to idle nodes to perform feature extraction operations based on the real-time data traffic and CPU utilization of each edge node.

10. A data processing system based on an artificial intelligence algorithm according to claim 9, characterized in that: The load balancer controller performs the following operations: when the data traffic of the edge node exceeds a preset peak, it redirects part of the data flow to the adjacent node, and maintains the temporal continuity of the dynamic feature encoding during the redirection process.

Images