A data processing method and system

Abstract

This invention discloses a data processing method and system, belonging to the field of data processing. The method includes: S1. In the initial operation phase of the system, collecting multi-source input data under the target business scenario, analyzing its statistical distribution, correlation, and time series characteristics, and forming a data feature baseline vector; S2. During operation, comparing the real-time feature vector with the data feature baseline vector, calculating the feature offset index and determining the offset trend, and generating a feature evolution signal when conditions are met; S3. Based on the feature evolution signal, enabling at least two data processing links with differences, processing the same dataset in parallel, and quantifying the processing effect; S4. Inputting the collaborative processing results and processing parameters into a feedback modeling module to construct and update a processing process-result mapping model; S5. Based on the mapping model, dynamically adjusting the subsequent processing links, parameters, and execution order, and recording the adjustment basis. Beneficial effects: Improving the overall processing performance and stability of the system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data processing method and system. Background Technology

[0002] In the field of data processing, existing technologies typically employ a process-oriented and modular approach. This involves data collection, preprocessing, computational analysis, and result output to achieve effective management and utilization of business data. This type of approach boasts a mature technological system, high operational stability, and maintains good processing efficiency and result consistency within given data scales and application scenarios. It offers significant advantages, particularly in environments with well-defined rules and relatively fixed data structures, including low implementation costs, short deployment cycles, and strong maintainability.

[0003] However, from a technological evolution perspective, existing data processing methods still harbor some limitations that are not easily identifiable. For example, their processing flows are mostly based on static design assumptions, implicitly assuming that data characteristics, correlations, and evolutionary trends remain relatively stable during long-term system operation. While this assumption may not reveal problems in the short term, it can limit further improvements in processing performance as data complexity gradually increases. Furthermore, existing methods typically optimize a single processing link, focusing on local efficiency or accuracy, lacking an inherent synergistic mechanism at the overall data value utilization level, thus constraining the system's performance ceiling without being easily detected. In addition, the utilization of feedback from the data processing process itself is relatively limited, preventing potential optimization space from naturally emerging within the existing technological framework. Summary of the Invention

[0004] The purpose of this invention is to provide a data processing method and system to address the problems mentioned in the background art: Their processing flows are mostly based on static design assumptions, implicitly assuming that data characteristics, correlations, and evolution trends remain relatively stable during long-term system operation. While this assumption may not reveal problems in the short term, it can limit further improvements in processing performance as data complexity gradually increases. Furthermore, existing methods typically optimize a single processing link, focusing on local efficiency or accuracy, lacking an inherent synergistic mechanism at the overall data value utilization level, thus constraining the system's performance ceiling without being easily detected. In addition, the utilization of feedback from the data processing process itself is relatively limited, preventing potential optimization space from naturally emerging within the existing technical framework.

[0005] Technical solution: A data processing method includes the following steps: S1. During the initial operation phase of the system, multi-source input data under the target business scenario is collected, and the statistical distribution characteristics, correlation structure and time series change characteristics of the multi-source input data are analyzed. A data feature baseline vector is formed by normalization and weighted integration to describe the implicit stability assumptions on which the current data processing depends. S2. During the data processing operation, the real-time input data feature vector is compared with the data feature baseline vector dimension by dimension in a periodic and triggered manner to calculate the comprehensive feature offset index, and the offset trend is determined according to the moving average, fluctuation range and continuous detection period; when the offset index continuously exceeds the preset threshold and the direction is consistent, a feature evolution signal is generated to indicate the continuous evolution of data features. S3. Based on the feature evolution signal, dynamically enable at least two data processing links with different processing strategies, wherein each link has differences in at least one of the feature selection rules, processing parameters and processing order, perform processing operations on the same dataset respectively, and uniformly quantify the efficiency index and quality index of each processing result to form a set of collaborative processing results. S4. Input the collaborative processing results and corresponding processing parameters into the feedback modeling module to construct a processing process-result mapping model. The processing process-result mapping model takes processing link selection information, link parameters and intermediate results as input and processing effect indicators as output. It is used to characterize the influence relationship of different processing strategies on the data value mining effect and continuously update the processing process-result mapping model to reflect the latest data. S5. Based on the process-result mapping model, the parameter configuration, processing link selection and execution order in the subsequent data processing flow are dynamically adjusted, and the adjustment content and basis are recorded, so as to continuously improve the overall processing performance without changing the external functional performance.

[0006] Preferably, the determination of the evolution trend of the S2 data features includes the following steps: S2-1. Classify the input data according to the data source and business type, and calculate the mean, variance, maximum value, minimum value, quantiles and correlation between features for each type of data; S2-2. Normalize each type of feature and integrate them according to preset weights to form a baseline feature vector; S2-3. Compare the real-time acquired feature vector with the baseline vector dimension by dimension and calculate the difference in each dimension; S2-4. The differences are weighted and summed to generate a comprehensive offset index, and the moving average and fluctuation range are calculated to exclude short-term anomalies and occasional fluctuations. S2-5. During a continuous detection cycle, when the offset index continuously exceeds the preset threshold and the direction is consistent, it is determined that the data features have a continuous evolution trend. S2-6. Mark the judgment results with relevant data dimensions and timestamps for subsequent processing and tracking.

[0007] Preferably, the S2 update of the data feature baseline includes the following steps: S2-7. Keep the baseline frozen during the current processing cycle to prevent frequent updates caused by short-term fluctuations; S2-8. Only locally update the feature dimensions that are determined to be in a continuous evolutionary trend; S2-9. Record the update dimensions, timestamps, and weighting parameters used to ensure traceability and reproducibility; S2-10. After the data feature baseline is updated, the latest baseline is used as a reference for the next cycle to ensure the accuracy of continuous evolution determination.

[0008] Preferably, S3 performs standardization processing on the intermediate outputs of each link and calculates the contribution index of each link in different feature dimensions. The processing results between links are weighted and fused based on the contribution index to form an optimized set of collaborative processing results, which are used for subsequent collaborative decision-making and performance evaluation.

[0009] Preferably, processing efficiency indicators and result quality indicators are calculated for the processing results of each processing link; The results of each link are weighed according to a comprehensive strategy to form a set of collaborative processing results, which not only includes the best results but also retains the second-best results.

[0010] Preferably, establishing a process-result mapping model includes the following steps: S4-1. Process link selection information, link parameters, and intermediate results as model inputs; S4-2. Use the processing effect indicators as model output; S4-3. Establish the mapping relationship between input and output to evaluate the impact of different processing strategies on the overall system performance; S4-4. Record the mapping relationship and the parameters of the processing-result mapping model to ensure reproducibility and operability.

[0011] Preferably, the selection of processing links, parameters, and performance indicators for each round are collected; the mapping relationship is updated in the mapping relationship and the processing process-result mapping model so that the mapping relationship and the processing process-result mapping model reflect the impact of the latest data on system performance; historical mapping records are saved for subsequent adaptive adjustments and performance evaluation.

[0012] Preferably, the adaptive optimization of subsequent processing in step S5 includes the following steps: S5-1. Based on the output of the process-result mapping model, evaluate the impact of each link and parameter on the overall performance; S5-2. Dynamically adjust the link selection, processing parameters, and execution order in subsequent data processing flows; S5-3. Record the adjustment content, timestamp, and basis for adjustment for performance monitoring and subsequent model updates.

[0013] Preferably, the storage medium stores computer-executable instructions that, when executed on a computing device, cause the device to implement the data processing method according to claims 1 to 8.

[0014] Preferably, the data processing system is designed according to one of the data processing methods described in 1-9, including a data acquisition module for collecting multi-source input data; a data processing module including a baseline construction module, a feature offset calculation module, an evolution determination module, and a multi-link collaborative processing module; a feedback modeling module for establishing and continuously updating a processing process-result mapping model; an adaptive optimization module for adjusting subsequent processing links and parameters; and a data storage module for storing processing results, offset indices, evolution determination information, and model parameters.

[0015] Compared with the prior art, the advantages of the present invention are as follows: (1) Construct a data feature baseline model based on multi-source input data to characterize the implicit stability assumption and improve the accuracy of initial processing; (2) Periodic and trigger-based determination of feature shift indicators, and identification of continuous evolution trends, to avoid performance limits under surface stability; (3) Dynamically enable at least two data processing links with different processing strategies to perform collaborative processing on the same dataset, thus breaking through the implicit constraints of a single link; (4) Quantify and calculate the contribution of intermediate outputs of each link to achieve weighted fusion between links and improve the collaborative processing effect; (5) Construct a process-result mapping model to map the processing parameters to the results, and use it to evaluate the impact of the strategy on the data value mining effect; (6) Adaptive adjustments are made based on the mapping model, including parameter configuration, link selection and execution order, to continuously optimize the overall processing performance; (7) Retain suboptimal processing results for collaborative and feedback modeling to improve system stability and robustness; (8) Record detailed information on the processing and adjustment operations to achieve traceable and reproducible closed-loop optimization. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall process of a data processing method according to the present invention; Figure 2 This is a schematic diagram of the overall modules of a data processing system according to the present invention. Detailed Implementation

[0017] For examples, please refer to Figures 1-2 A data processing method includes the following steps: S1. During the initial operation phase of the system, multi-source input data under the target business scenario is collected. The statistical distribution characteristics, correlation structure and time series change characteristics of the multi-source input data are analyzed. The data feature baseline vector is formed by normalization and weighted integration to describe the implicit stability assumptions on which the current data processing depends. S2. During the data processing operation, the real-time input data feature vector is compared with the data feature baseline vector dimension by dimension in a periodic and triggered manner to calculate the comprehensive feature offset index, and the offset trend is determined based on the moving average, fluctuation range and continuous detection period; when the offset index continuously exceeds the preset threshold and the direction is consistent, a feature evolution signal is generated to indicate the continuous evolution of data features. Specifically, during the data processing operation, when comparing the feature vector of the real-time input data with the baseline vector dimension by dimension, the following operations are performed: The difference in each feature dimension is calculated by subtracting the baseline value from the real-time value, and then summing the values ​​with weights to obtain the comprehensive offset index. The moving average can be calculated using a sliding window method (e.g., the past 5 to 10 sampling points), and the fluctuation range is used to remove outliers. Outlier removal methods can use Z-score judgment or upper and lower limit threshold filtering. If the offset index continuously exceeds the preset threshold (e.g., a change of 10% to 20% in the example) and the direction is consistent, the number of consecutive periods can be 3 to 5 samplings in the example. The time window is adjusted according to the business scenario, such as a sampling period of 1 hour, 1 day, or 1 week.

[0018] Periodic determinations can be performed at fixed time intervals (every 5 minutes or every hour); triggered determinations can be triggered by external events, such as business anomalies, sudden increases in data volume, or degradation of link performance.

[0019] S3. Based on the feature evolution signal, dynamically enable at least two data processing links with different processing strategies. Each link has differences in at least one of the feature selection rules, processing parameters, and processing order. Perform processing operations on the same dataset separately, and uniformly quantify the efficiency and quality indicators of each processing result to form a set of collaborative processing results. Specifically, after the feature evolution signal is triggered, multi-link processing is performed as follows: Each link differs in at least one of the following: feature selection rules (selecting different feature subsets), processing parameters (such as weight coefficients and threshold parameters), or processing order (processing high-priority features first or low-priority features first); Efficiency metrics include processing latency, resource utilization, or throughput; quality metrics include processing accuracy, error rate, or information fidelity; each metric can be standardized and quantified (normalized from 0 to 1) for comparison; the processing results of each link are weighted and fused, and the weights can be calculated based on the contribution index (the contribution of each feature to the overall processing effect); The final set of collaborative results is formed, while the best and second-best results are retained for subsequent analysis or backtracking.

[0020] S4. Input the collaborative processing results and corresponding processing parameters into the feedback modeling module to construct a processing process-result mapping model. The processing process-result mapping model takes the processing link selection information, link parameters and intermediate results as inputs and the processing effect indicators as outputs. It is used to characterize the impact of different processing strategies on the data value mining effect and continuously update the processing process-result mapping model to reflect the latest data. Specifically, the model inputs are: link selection information, processing parameters, and intermediate result vectors; the output is: a vector of processing performance indicators (efficiency indicators and quality indicators). The model can be implemented using table mapping, statistical modeling (such as multiple linear regression), or simple algorithms; the model generates a table of correspondence between links and result indicators for rapid evaluation of strategy performance. Use rolling updates (replacing the oldest historical data), incremental updates (updating only new data contributions), or batch updates (periodicly rebuilding the model). Each update records the model parameters, input-output mapping, and update time to ensure traceability and reproducibility.

[0021] S5. Based on the process-result mapping model, the parameter configuration, processing link selection and execution order in the subsequent data processing flow are dynamically adjusted, and the adjustment content and basis are recorded, so as to continuously improve the overall processing performance without changing the external functional performance.

[0022] Specifically, adjustments are triggered based on the deviation of the model output from the target performance index threshold; the optimal combination or weights are selected or adjusted based on the link contribution index to ensure continuous improvement in system performance; and the processing links are dynamically selected: high-contribution links are prioritized. Parameter adjustments: such as weights, thresholds, or feature subset size; Execution order adjustment: Adjust the feature processing order or the chain execution order to optimize overall processing performance; The adjusted content, timestamp, and basis for the adjustment are recorded in logs, databases, or storage structures; It can be used for performance monitoring, subsequent model updates, or backtracking analysis to ensure that closed-loop optimization is feasible.

[0023] Determining the evolution trend of S2 data features includes the following steps: S2-1. Classify the input data according to the data source and business type, and calculate the mean, variance, maximum value, minimum value, quantiles and correlation between features for each type of data; Specifically, input data can be categorized by source, such as sensor A / B, business log system 1 / 2, external data interface, etc.; or by business type, such as transaction data, user behavior data, device status data; each type of data forms an independent feature set, and each feature set is used to calculate statistical features (mean, variance, maximum value, minimum value, quantile) and correlation between features separately; The classification criteria can be exemplified in the instruction manual as follows: data from the same source and of the same business type belong to the same category.

[0024] S2-2. Normalize each type of feature and integrate them according to preset weights to form a baseline feature vector; S2-3. Compare the real-time acquired feature vector with the baseline vector dimension by dimension and calculate the difference in each dimension; S2-4. The differences are weighted and summed to generate a comprehensive offset index, and the moving average and fluctuation range are calculated to exclude short-term anomalies and occasional fluctuations. Specifically, the initial weights are based on an assessment of the importance of historical data (e.g., the influence of features on the results, the size of the variance, and the strength of the correlation). The weights can be normalized so that the sum of all feature weights is 1, ensuring consistency and reproducibility of the calculation; Update strategy example: For feature dimensions with continuous evolution trends, the weight can be appropriately increased to emphasize their impact on the comprehensive offset index; for short-term abnormal or stable features, the weight can be reduced to avoid noise introduced by frequent updates; weight updates can be recorded at each baseline update to ensure traceability; when calculating the comprehensive offset index, the difference of each dimension is multiplied by the corresponding weight, and then the weighted sum is obtained to form the final index. Moving averages and fluctuation ranges can be combined with weight adjustments, such as using a smoother sliding window for highly weighted features.

[0025] S2-5. During a continuous detection cycle, when the offset index continuously exceeds the preset threshold and the direction is consistent, it is determined that the data features have a continuous evolution trend. Specifically, the judgment logic for offset indicators exceeding the threshold consecutively is combined with a comprehensive indicator after weighting; the number of consecutive periods is 3 to 5 sampling periods, and the threshold range can be preset according to historical statistical characteristics; the time window is set according to the business processing frequency (5 minutes / hour / day).

[0026] S2-6. Mark the judgment results with relevant data dimensions and timestamps for subsequent processing and tracking.

[0027] The S2 update data feature baseline includes the following steps: S2-7. Keep the baseline frozen during the current processing cycle to prevent frequent updates caused by short-term fluctuations; Specifically, within the current cycle, feature dimensions that do not show a sustained evolution trend will have their baselines frozen to avoid frequent adjustments due to short-term fluctuations; only feature dimensions that are determined to have a sustained evolution trend will be locally updated, and the new observations will be included in the baseline calculation. Partial updates can be performed using a weighted average method: New baseline = α × New observation features + (1-α) × Old baseline α is the update coefficient, with an example of 0.2 to 0.5, which can be adjusted according to business sensitivity and data volatility.

[0028] S2-8. Only locally update the feature dimensions that are determined to be in a continuous evolutionary trend; S2-9. Record the update dimensions, timestamps, and weighting parameters used to ensure traceability and reproducibility; Specifically, each baseline update records the following information: update dimension, timestamp, weight value used, and update coefficient α; the latest baseline is used as a reference for the next cycle after the update is completed to ensure that the continuous evolution judgment can be reproduced; the example storage method in the manual is: database table, log file or memory structure to ensure that historical data is traceable.

[0029] S2-10. After the data feature baseline is updated, the latest baseline is used as the reference for the next cycle to ensure the accuracy of continuous evolution determination.

[0030] S3 standardizes the intermediate outputs of each link and calculates the contribution index of each link in different feature dimensions. The processing results between links are weighted and fused based on the contribution index to form an optimized set of collaborative processing results, which are used for subsequent collaborative decision-making and performance evaluation.

[0031] Specifically, intermediate output standardization Intermediate output for each feature dimension i of each link l Normalization is performed: in: This represents the original processed output value of link l in feature dimension i; The minimum value of feature dimension i across all link history outputs; The maximum value of feature dimension i across all link history outputs; This is the normalized output value, ranging from [0,1].

[0032] Specifically, the contribution metric for each processing link is obtained by weighting and summing its standardized outputs across multiple feature dimensions: in, This is the contribution metric for the l-th processing link; n is the number of feature dimensions; The weight of the i-th feature dimension reflects the importance of that feature to the processing result.

[0033] Specifically, based on the contribution index of each processing link, the processing results of each link are weighted and fused to obtain the collaborative processing result: in: represents the collaborative processing result for the i-th feature dimension; m represents the number of links participating in the collaborative processing. Let be the fusion weight of the j-th processing link, and satisfy . Fusion weights Based on the contribution index of the corresponding link Sure.

[0034] Calculate the processing efficiency index and result quality index for the processing results of each processing link; The results of each link are weighed according to a comprehensive strategy to form a set of collaborative processing results, which not only includes the best results but also retains the second-best results.

[0035] Specifically, processing efficiency metrics are used to characterize the ability of a processing chain to complete data processing per unit of time, including at least one of processing time or throughput. Results quality metrics: used to characterize the effectiveness or accuracy of the processing results, including at least one of accuracy, completeness or consistency.

[0036] Specifically, the processing results of each processing link are comprehensively evaluated based on their processing efficiency and result quality indicators. Under the premise of meeting the minimum quality threshold, the processing result with the highest comprehensive evaluation value is selected as the main output. When forming the collaborative processing result set, in addition to the main output, processing results with the second-best comprehensive evaluation value that meet the preset quality requirements are also retained. Suboptimal results are used for subsequent feedback modeling, adaptive adjustment, or anomaly comparison analysis.

[0037] Establishing a process-result mapping model includes the following steps: S4-1. Process link selection information, link parameters, and intermediate results as model inputs; S4-2. Use the processing effect indicators as model output; S4-3. Establish the mapping relationship between input and output to evaluate the impact of different processing strategies on the overall system performance; S4-4. Record the mapping relationship and processing procedure - result mapping model parameters to ensure reproducibility and operability.

[0038] Collect the selection of processing links, parameters, and performance indicators for each round; update the mapping relationship in the mapping relationship and the processing process-result mapping model so that the mapping relationship and the processing process-result mapping model reflect the impact of the latest data on system performance; save historical mapping records for subsequent adaptive adjustments and performance evaluation.

[0039] Specifically, the process-result mapping model is used to describe the impact of different processing links and their parameter configurations on the processing effect.

[0040] After each round of data processing is completed, the actual processing link selection information, the corresponding processing parameter configuration, and the intermediate results generated during processing are used as input information, and the processing efficiency index and result quality index are used as output information. The correspondence between input and output is established in a structured way, thereby forming a processing process-result mapping relationship.

[0041] The mapping relationships are stored in the form of a record table. Each mapping record corresponds to a complete data processing process and includes at least a link identifier, parameter information, intermediate result summary, processing effect indicators, and generation timestamp. This method establishes a traceable correspondence between different processing strategies and their processing effects.

[0042] During subsequent data processing, the system continuously collects the link selection, parameter configuration, and processing performance indicators corresponding to each round of processing, and generates new mapping records based on the original mapping relationships, thereby updating the mapping relationships. The updates are performed incrementally, without overwriting existing mapping records, to ensure the integrity of historical data.

[0043] When making subsequent adaptive adjustments or performance evaluations, decisions are made based primarily on the latest generated mapping records, while retaining the ability to query historical mapping records to support trend analysis and comparative evaluation. This approach ensures that the process-result mapping model remains synchronized with the latest data and possesses good reproducibility and operability.

[0044] When the number of mapping records reaches a preset threshold, historical mapping records are managed in chronological order of their generation, and earlier records are archived and stored, thereby controlling the storage scale while ensuring model continuity.

[0045] S5 performs adaptive optimization for subsequent processing, including the following steps: S5-1. Based on the output of the process-result mapping model, evaluate the impact of each link and parameter on the overall performance; S5-2. Dynamically adjust the link selection, processing parameters, and execution order in subsequent data processing flows; S5-3. Record the adjustment content, timestamp, and basis for adjustment for performance monitoring and subsequent model updates.

[0046] Specifically, the output of the process-result mapping model is used to reflect the degree of influence of different processing links and their parameter configurations on the processing effect.

[0047] Based on the latest mapping results recorded in the mapping model, the system compares and evaluates the processing effect indicators corresponding to each processing link and key parameters, and determines the processing strategies that contribute more or less to the overall performance improvement.

[0048] When the mapping model indicates that a certain processing link or parameter configuration has a positive or negative impact on the processing effect in consecutive processing rounds, it serves as the basis for subsequent adaptive adjustments, guiding the direction of link selection and parameter adjustment.

[0049] Specifically, the adaptive adjustment is limited to a pre-defined adjustable range, which includes at least: Enable or disable the link; Adjusting the values ​​of processing parameters; Adjust the execution order of multiple processing links.

[0050] The processing parameters are adjusted gradually, with each adjustment not exceeding a portion of the preset adjustment range for the corresponding parameter, in order to avoid abrupt changes in the processing results.

[0051] The adjustment range is set during system initialization or runtime configuration and remains unchanged or is updated as needed during operation.

[0052] Specifically, to prevent frequent or excessive adjustments from causing system performance fluctuations, at least one of the following control measures should be adopted during the adaptive adjustment process: A minimum interval processing period is set between two consecutive adjustments; no new adjustment is executed if the period is not reached. If continuous adjustments do not improve the processing effect, pause subsequent adjustments and maintain the current processing configuration; When the fluctuation of the processing effect index is detected to exceed the preset range, it will revert to the previous stable processing link and parameter configuration.

[0053] Through the above control measures, we can ensure that the adaptive adjustment process improves overall processing performance while maintaining the stability of system operation.

[0054] Specifically, after each adaptive adjustment, the system records the corresponding adjustment content, adjustment timestamp, and the output information of the mapping model on which it was based. The records are used for updates to the subsequent processing-result mapping model, thus forming a complete closed-loop optimization mechanism.

[0055] The storage medium stores computer-executable instructions, which, when executed on a computing device, cause the device to implement a data processing method according to claims 1 to 8.

[0056] Specifically, the aforementioned computer-executable instructions are stored in the form of modular instruction sets, with each type of instruction corresponding to a functional operation, including at least: data acquisition and feature extraction instructions; data feature baseline construction and update instructions; feature offset calculation and evolution determination instructions; multi-link processing and collaborative fusion instructions; mapping relationship recording and update instructions; and adaptive adjustment control instructions.

[0057] Various instructions are executed through sequential or conditional calls, enabling the computing device to run in sequence according to the steps of the data processing method.

[0058] Specifically, computer-executable instructions run on general-purpose or special-purpose computing devices, which include at least a processor and memory. The processor is used to execute instructions stored in the storage medium, and the memory is used to store intermediate data, feature vectors and processing results during instruction execution.

[0059] The execution environment can be a local computing environment, an edge computing environment, or a cloud computing environment, and is not limited by a specific hardware platform or operating system.

[0060] Specifically, at least one of the following data structures must be used during instruction execution: Feature vector data structure, used to store statistical feature values ​​of multi-source input data; The mapping record data structure is used to store the correspondence between processing links, parameter configurations, and processing performance indicators; Adjust the record data structure to store the content and time information of each adaptive adjustment.

[0061] The above data structure enables technicians to directly implement the corresponding instruction logic based on the contents of the instruction manual.

[0062] Specifically, the storage medium can be a non-volatile storage medium or a volatile storage medium, including at least one of the following: solid-state storage medium; magnetic storage medium; semiconductor memory; cloud storage medium.

[0063] The specific type of storage medium does not constitute a limitation of this invention.

[0064] The data processing system is designed based on one of the data processing methods in 1-9, and includes a data acquisition module for collecting multi-source input data; a data processing module, including a baseline construction module, a feature offset calculation module, an evolution determination module, and a multi-link collaborative processing module; a feedback modeling module for establishing and continuously updating a processing process-result mapping model; an adaptive optimization module for adjusting subsequent processing links and parameters; and a data storage module for storing processing results, offset indices, evolution determination information, and model parameters.

[0065] Specifically, the data acquisition module is used to collect multi-source input data and convert the data into raw feature data in a unified format, which is then transmitted to the data processing module through the data interface. The data processing module receives the raw feature data, sequentially completes the construction of data feature baseline, feature offset calculation, evolution trend determination and multi-link collaborative processing, and generates the processing results; The multi-link collaborative processing module transmits the processing results and corresponding parameters of each link to the feedback modeling module; After the feedback modeling module generates or updates the process-result mapping model, it sends the relevant evaluation results to the adaptive optimization module. The adaptive optimization module generates adjustment instructions based on the evaluation results and feeds them back to the data processing module for subsequent processing adjustments. The data storage module maintains data communication with the above modules and is used to store various types of data generated during the processing.

[0066] Specifically, after completing a round of mapping relationship updates, the feedback modeling module sends an update completion signal and the corresponding evaluation results to the adaptive optimization module. When the evaluation results meet the preset adjustment conditions, the adaptive optimization module triggers the adjustment logic and generates the corresponding adjustment parameters or link selection information.

[0067] Specifically, the baseline construction module uses statistical distribution analysis to calculate the mean, variance, and correlation characteristics of the data; the feature offset calculation module evaluates the offset based on the difference between real-time features and baseline features; the multi-link collaborative processing module integrates the output results of different processing links using a weighted fusion strategy; and the adaptive optimization module adjusts the subsequent processing strategy based on the historical records in the mapping relationship.

[0068] The above implementation methods are merely examples and do not constitute a limitation on the specific implementation methods of the module.

[0069] Specifically, the data storage module adopts a structured storage method, which stores at least the following information: processing results and intermediate outputs of each processing link; feature offset indicators and evolution judgment results; parameters of the processing-result mapping model and historical mapping records; adaptive adjustment records and corresponding timestamps. Through the above storage method, the traceability and reproducibility of various types of data during system operation are ensured.

[0070] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data processing method, characterized in that, The data processing method includes the following steps: S1. During the initial operation phase of the system, multi-source input data under the target business scenario is collected, and the statistical distribution characteristics, correlation structure and time series change characteristics of the multi-source input data are analyzed. A data feature baseline vector is formed by normalization and weighted integration to describe the implicit stability assumptions on which the current data processing depends. S2. During the data processing operation, the real-time input data feature vector is compared with the data feature baseline vector dimension by dimension in a periodic and triggered manner to calculate the comprehensive feature offset index, and the offset trend is determined according to the moving average, fluctuation range and continuous detection period; when the offset index continuously exceeds the preset threshold and the direction is consistent, a feature evolution signal is generated to indicate the continuous evolution of data features. S3. Based on the feature evolution signal, dynamically enable at least two data processing links with different processing strategies, wherein each link has differences in at least one of the feature selection rules, processing parameters and processing order, perform processing operations on the same dataset respectively, and uniformly quantify the efficiency index and quality index of each processing result to form a set of collaborative processing results. S4. Input the collaborative processing results and corresponding processing parameters into the feedback modeling module to construct a processing process-result mapping model. The processing process-result mapping model takes processing link selection information, link parameters and intermediate results as input and processing effect indicators as output. It is used to characterize the influence relationship of different processing strategies on the data value mining effect and continuously update the processing process-result mapping model to reflect the latest data. S5. Based on the process-result mapping model, the parameter configuration, processing link selection and execution order in the subsequent data processing flow are dynamically adjusted, and the adjustment content and basis are recorded, so as to continuously improve the overall processing performance without changing the external functional performance.

2. The data processing method according to claim 1, characterized in that, The determination of the evolution trend of S2 data features includes the following steps: S2-1. Classify the input data according to the data source and business type, and calculate the mean, variance, maximum value, minimum value, quantiles and correlation between features for each type of data; S2-2. Normalize each type of feature and integrate them according to preset weights to form a baseline feature vector; S2-3. Compare the real-time acquired feature vector with the baseline vector dimension by dimension and calculate the difference in each dimension; S2-4. The differences are weighted and summed to generate a comprehensive offset index, and the moving average and fluctuation range are calculated to exclude short-term anomalies and occasional fluctuations. S2-5. During a continuous detection cycle, when the offset index continuously exceeds the preset threshold and the direction is consistent, it is determined that the data features have a continuous evolution trend. S2-6. Mark the judgment results with relevant data dimensions and timestamps for subsequent processing and tracking.

3. The data processing method according to claim 1, characterized in that, The S2 update data feature baseline includes the following steps: S2-7. Keep the baseline frozen during the current processing cycle to prevent frequent updates caused by short-term fluctuations; S2-8. Only locally update the feature dimensions that are determined to be in a continuous evolutionary trend; S2-9. Record the update dimensions, timestamps, and weighting parameters used to ensure traceability and reproducibility; S2-10. After the data feature baseline is updated, the latest baseline is used as a reference for the next cycle to ensure the accuracy of continuous evolution determination.

4. The data processing method according to claim 1, characterized in that, S3 standardizes the intermediate outputs of each link and calculates the contribution index of each link in different feature dimensions. The processing results between links are weighted and fused based on the contribution index to form an optimized set of collaborative processing results, which are used for subsequent collaborative decision-making and performance evaluation.

5. The data processing method according to claim 4, characterized in that, Calculate the processing efficiency index and result quality index for the processing results of each processing link; The results of each link are weighed according to a comprehensive strategy to form a set of collaborative processing results, which not only includes the best results but also retains the second-best results.

6. The data processing method according to claim 1, characterized in that, Establishing a process-result mapping model includes the following steps: S4-1. Process link selection information, link parameters, and intermediate results as model inputs; S4-2. Use the processing effect indicators as model output; S4-3. Establish the mapping relationship between input and output to evaluate the impact of different processing strategies on the overall system performance; S4-4. Record the mapping relationship and the parameters of the processing-result mapping model to ensure reproducibility and operability.

7. The data processing method according to claim 6, characterized in that, Collect the processing link selection, parameters, and performance indicators for each round; update the mapping relationship in the mapping relationship and the processing process-result mapping model so that the mapping relationship and the processing process-result mapping model reflect the impact of the latest data on system performance; save historical mapping records for subsequent adaptive adjustments and performance evaluation.

8. The data processing method according to claim 1, characterized in that, The adaptive optimization of subsequent processing in S5 includes the following steps: S5-1. Based on the output of the process-result mapping model, evaluate the impact of each link and parameter on the overall performance; S5-2. Dynamically adjust the link selection, processing parameters, and execution order in subsequent data processing flows; S5-3. Record the adjustment content, timestamp, and basis for adjustment for performance monitoring and subsequent model updates.

9. The data processing method according to claim 1, characterized in that, The storage medium stores computer-executable instructions that, when executed on a computing device, cause the device to implement the data processing method as described in claims 1 to 8.

10. A data processing system, characterized in that, The data processing system is designed according to a data processing method described in 1-9, and includes a data acquisition module for collecting multi-source input data; and a data processing module including a baseline construction module, a feature offset calculation module, an evolution determination module, and a multi-link collaborative processing module. The feedback modeling module is used to build and continuously update the process-result mapping model. The adaptive optimization module is used to adjust subsequent processing links and parameters; The data storage module is used to store processing results, offset indices, evolution determination information, and model parameters.

Images