Software development application data processing method based on AI large model

Abstract

The invention discloses a software development application data processing method based on an AI large model, and relates to the technical field of data processing and streaming computation.According to the software development application data processing method based on the AI large model, spatial and temporal features are extracted by means of a pre-trained Transform model, time dependence and spatial association in data streams can be captured more accurately in combination with a self-attention mechanism, and the data processing efficiency is improved. Therefore, the accuracy and real-time performance of anomaly detection are improved, and misjudgment or omission caused by environmental interference or equipment faults is reduced. Secondly, according to the self-adaptive threshold mechanism, by dynamically updating a threshold and adjusting an abnormal score based on statistics in a window, the system can flexibly adapt to the change of data distribution, and the problem of response lag occurring in a high-speed data stream in a traditional fixed threshold method is avoided; besides, a reinforcement learning strategy is introduced, the error, the time delay and the service constraint are comprehensively reconstructed through a state space and a reward function, and a targeted repair action is generated, so that the self-adaptive capability of the system in a complex scene is enhanced, and the manual intervention requirement is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing and streaming computing technology, and in particular to a data processing method for software development applications based on large AI models. Background Technology

[0002] In IoT, financial transactions, and cloud-native systems, data is generated at high speed and continuously, and business requirements necessitate computation and decision-making with low latency.

[0003] While existing stream processing systems can perform outlier detection based on sliding windows and threshold rules, and combine simple repair methods such as interpolation / filtering, they often encounter problems such as detection delays, increased false positives and false negatives, and distorted repair when faced with scenarios such as schema drift, out-of-order / late events, and concept drift.

[0004] In recent years, unsupervised models (such as isolated forests and autoencoders) and online learning methods have improved adaptability to some extent. However, under engineering constraints such as event temporal semantics / water level processing, back pressure, and end-to-end consistency, detection and repair are difficult to form an interpretable, traceable, and online-evolvable closed loop, making it difficult to simultaneously meet the service level requirements of P95 / P99 latency and accuracy. Therefore, there is still an urgent need for an integrated anomaly detection and repair mechanism that can achieve context-awareness for multi-source spatiotemporal related data while maintaining low overhead and consistency guarantees. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] This invention provides a data processing method for software development applications based on AI large models to solve the problems of detection delay, rigid repair strategies, and inability to adapt to dynamic data changes in real-time data stream anomaly detection, which leads to insufficient reliability of downstream applications.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] This invention provides a data processing method for software development applications based on a large AI model, comprising:

[0009] Step S1: Obtain the real-time data stream and perform windowing processing on the real-time data stream based on the event time;

[0010] Step S2: Encode the windowed data using a pre-trained Transformer model to obtain spatiotemporal features that represent time dependence and spatial correlation.

[0011] Step S3: Calculate the anomaly score for each data point based on the spatiotemporal features, and generate an anomaly mask according to the adaptive threshold;

[0012] Step S4: Input the anomaly mask and the spatiotemporal features into the reinforcement learning strategy to generate repair actions for the anomaly points;

[0013] Step S5: Perform the repair action to repair the real-time data stream, obtain the repaired data stream, and record the repair metadata;

[0014] Step S6: Output the repaired data stream and the repaired metadata for downstream applications to consume.

[0015] As a preferred embodiment of the data processing method for software development applications based on AI large models described in this invention, the method of acquiring real-time data streams includes receiving data from IoT devices, financial transaction systems, or cloud-native applications, and the windowing processing includes sliding windows or scrolling windows based on event time, combined with water level lines to handle late and out-of-order data.

[0016] As a preferred embodiment of the data processing method for software development applications based on AI large models described in this invention, the anomaly score includes a weighted combination of contextual inconsistency based on self-attention weights and deviation based on reconstruction error.

[0017] The adaptive threshold is dynamically updated based on statistics within the window.

[0018] The calculation definition for the weighted combination of the anomaly scores and the dynamic update of the adaptive threshold is as follows:

[0019] Step E1, in window number... Within the event time window, for the first For each data point, the context inconsistency and reconstruction bias are linearly synthesized after robust normalization to obtain the score:

[0020] ,

[0021] in, For the first Anomaly scores for each data point This is a normalized value for the degree of context inconsistency. To reconstruct the normalized value of the deviation, For window The weighting coefficients, Indexing data points within the window, The window number;

[0022] Normalize:

[0023] ,

[0024] in, For measurement The normalization result, when have to ,when have to , For the original measurement, For window The sample set for this metric, For the median operator, For the median absolute deviation operator, It is a scaling constant. It is the numerical stability constant;

[0025] Perform weight updates:

[0026] , , ,

[0027] in, For window The weighting coefficients, These are the reliability measures for the two types of evidence, and For window The two types of primitive metric sets;

[0028] Step E2, evaluate the rating set within the window. Calculate the robust threshold and update it over time:

[0029] ,

[0030] in, For window The threshold, The threshold of the previous window, This is the time smoothing coefficient. For set At the upper quantile level quantiles at , For quantile level parameters, This is the margin coefficient. The median absolute deviation of the rating set is used; the ratings are then mapped to anomaly masks.

[0031] ,

[0032] in, For the first Anomaly indication for each data point, For indicator functions, Rate this point. This is the current threshold.

[0033] Step E3, when the water level line causes late data to be incorporated into the window. When making up the calculation, only the affected ones Perform an incremental update, then recalculate according to step E2. With the mask, there is no need to rescan the already stable window.

[0034] As a preferred embodiment of the software development application data processing method based on AI large model described in this invention, the input of the Transformer model includes spatial embedding of time location encoding and device / geographic / topological identifiers, and the encoder extracts cross-time and cross-device correlations through multi-head self-attention.

[0035] As a preferred embodiment of the data processing method for software development applications based on AI large models described in this invention, the abnormal points are labeled with types based on the abnormal scores and context features, and the types include at least one or more of data quality abnormalities and statistical abnormalities.

[0036] As a preferred embodiment of the data processing method for software development applications based on AI large models described in this invention, the state of the reinforcement learning strategy includes the spatiotemporal features, anomaly mask, and repair history.

[0037] The action space includes interpolation, replacement, deletion, and their parameterized variants;

[0038] The reward function integrates reconstruction error, latency penalty, and business constraints to ensure consistency.

[0039] The steps for multi-objective reward aggregation and policy optimization in the reward function are as follows:

[0040] Step F1 aggregates the refactoring error, latency penalty, and business constraint consistency into an immediate reward, treating each event time window as a step, and then... Status and actions receive immediate rewards:

[0041] ,

[0042] in, For the first Instant rewards for each step To reconstruct the normalized value of the error, This is the normalized value of the delay. Aggregated penalties for violations of business constraints The weights of the three terms are given, and their sum is given. , The window number;

[0043] Perform error normalization:

[0044] ,

[0045] in, For the first Reconstruction error of step, For the recent error sample set, The median, This represents the median absolute deviation. It is the numerical stability constant;

[0046] Perform time delay normalization:

[0047] ,

[0048] in, The additional end-to-end latency caused by this repair step For business target latency threshold, For recent delay sets A robust scale, To obtain the largest operator;

[0049] Define the penalties for constraint violations:

[0050] ,

[0051] in, To constrain the number of entries, For the first The power to impose penalties under the constraints of the law For hinge operators, For the first Business constraint functions, For the penalty, For the first Step state, This is the repair action for this step;

[0052] Define adaptive weights:

[0053] ,

[0054] in, For the first Item weight, For robust reliability measurement, For the corresponding historical sequence;

[0055] Step F2, policy update and stabilization based on policy gradient, using actor-commentator update with baseline:

[0056] ,

[0057] in, For strategy parameters, For the policy learning rate, For the probability of a conditional action, For the advantages of time-difference, The entropy coefficient, For Shannon entropy, Let be the value function, with parameters as follows: , As a discount factor, Indicates by threshold Operators for global norm clipping;

[0058] In the formula: value parameters are updated using least squares. ,in For the value learning rate, gradients are pruned using the global norm, with a threshold of [value missing]. Introducing target value network parameters Perform exponential sliding: ,in The sliding coefficient is used for sampling, and the nearest sampling is adopted. Small batches in time order within a window, with entropy terms suppressing premature convergence;

[0059] Step F3, for auditing purposes, and When hot updates are performed in the configuration center and drift occurs, the trigger logic in step E2 will trigger a rollback or gray-scale deployment.

[0060] As a preferred embodiment of the data processing method for software development applications based on AI large models described in this invention, the repair execution includes repairing anomalies point by point or in segments, and generating repair metadata containing repair location, action type, parameters and confidence level; the output end supports idempotent writing and replayable recalculation.

[0061] As a preferred embodiment of the data processing method for software development applications based on a large AI model as described in this invention, the method involves: triggering online updates of the Transformer model and the reinforcement learning strategy based on drift detection results, wherein the online updates employ canary releases and retain rollback paths.

[0062] As a preferred embodiment of the software development application data processing method based on AI large model described in this invention, the output of step S6 includes writing the repaired data stream and the repaired metadata into the downstream topic or operator of the stream processing framework, and providing anomaly mask and confidence level to the downstream.

[0063] As a preferred embodiment of the data processing method for software development applications based on AI large models described in this invention, the method outputs interpretable information for the data points to be repaired, including attention contribution, anomaly score decomposition, and the basis for selecting repair actions.

[0064] The beneficial effects of this invention are as follows: First, by utilizing a pre-trained Transformer model to extract spatiotemporal features and combining it with a self-attention mechanism, this invention can more accurately capture the temporal dependencies and spatial correlations in the data stream, thereby improving the accuracy and real-time performance of anomaly detection and reducing misjudgments or omissions caused by environmental interference or equipment failure. Second, the adaptive threshold mechanism dynamically updates the threshold and adjusts the anomaly score based on in-window statistics, enabling the system to flexibly adapt to changes in data distribution and avoiding the response lag problem that occurs in traditional fixed threshold methods in high-speed data streams. Furthermore, the introduction of reinforcement learning strategies, through the comprehensive reconstruction of errors, latency, and business constraints using state space and reward functions, generates targeted repair actions, such as interpolation or replacement. This not only optimizes repair efficiency but also enhances the system's adaptability in complex scenarios, reducing the need for manual intervention. Simultaneously, the output of repair metadata and interpretable information, including anomaly score decomposition and action selection criteria, provides transparent traceability support for downstream applications, facilitating auditing and fault analysis, and further improving the reliability of data processing.

[0065] While maintaining real-time performance, this invention achieves collaborative optimization of detection and repair through AI-driven processes, significantly improving the robustness and maintainability of data stream processing, and is suitable for high-requirement scenarios such as the Internet of Things and financial transactions. Attached Figure Description

[0066] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0067] Figure 1 This is a flowchart illustrating the data processing method for software development applications based on AI large models in the embodiments. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0070] For example, the terms “first” and “second” used in this application are only used to distinguish and describe similar objects, to differentiate the first object from another object, and are not used to describe a specific order or sequence, nor should they be interpreted as indicating or implying relative importance.

[0071] This application proposes a data processing method for software development applications based on large AI models, combined with... Figure 1 As shown, the method includes:

[0072] Step S1: Acquire the real-time data stream and perform windowing processing on the real-time data stream based on event time. In this embodiment, event time refers to the source-side acquisition timestamp that arrives with the data, which is used for window division after correction for time zone and time synchronization deviation. Windowing is grouped by data source identifier or business primary key to avoid cross-source interference. Both sliding and scrolling modes are based on event time. The water level is the maximum observed event time minus the allowable delay, used to determine whether the window can be opened. The default window length is 5s, the sliding step size is 1s, and the allowable delay is 2 seconds. Based on the experience of verifying the processing delay of the link in the order of 100ms and the clock jitter of the device side in the range of ±1s, the value of s can be adjusted in the range of 1s~60s, 200ms~10s, and 0~30s. When the data lacks a valid event time or the time synchronization fails, it will be preferentially backed to the receiving time and the source will be marked. If the short-term out-of-order ratio exceeds 5% or there are fewer than 10 valid samples in the window, the allowable delay will be temporarily increased to twice the original value to ensure the sample size in the window. When resources are tight, the sliding step size can be optionally increased to 2s to reduce the frequency of status updates.

[0073] Step S2 involves encoding the windowed data using a pre-trained Transformer model to obtain spatiotemporal features representing time dependence and spatial correlation. For example, the pre-trained model is trained unsupervised or self-supervised based on historical data. The input includes normalized numerical features and encoded category / state bits. The temporal location encoding adopts a fixed or learnable method, and the spatial embedding is obtained by mapping device, geographic, or topological identifiers. The default hidden dimension is 128, the number of layers is 3, the number of attention heads is 4, and the dropout rate is 0.1, with ranges of 64~512, 2~6, 2~8, and 0~0.3, respectively. This is determined based on a trade-off between online P95 latency of less than 50ms and AUC improvement. When spatial identifiers are missing, they are replaced with zero vectors, and missing indicator bits are added to the features. The spatial embedding branch can be optionally closed to reduce inference overhead. When inference resources are insufficient or the queue length exceeds the upper limit, the hidden dimension is temporarily reduced to 64 or the last layer is closed to ensure the latency target.

[0074] Step S3: Calculate the anomaly score for each data point based on spatiotemporal features, and generate an anomaly mask based on an adaptive threshold;

[0075] Step S4: Input the anomaly mask and spatiotemporal features into the reinforcement learning policy to generate repair actions for the anomalies.

[0076] In this embodiment, the strategy input consists of the spatiotemporal characteristics of the current window, the corresponding anomaly indication, and the repair summary of the previous window. The output is the interpolation, replacement, or deletion of the marked points and their parameters. The default strategy makes a decision once per window as the time step. The action parameters are constrained by the minimum and maximum values ​​of the effective neighborhood in the same window within the numerical range. Interpolation prioritizes the use of adjacent effective points of the same source and dimension. When there are no available points in the neighborhood or the anomaly density exceeds 50%, robust statistics of the same group are temporarily used for replacement to avoid unstable interpolation.

[0077] Step S5: Perform the repair action to repair the real-time data stream, obtain the repaired data stream, and record the repair metadata;

[0078] For example, the repair metadata includes the primary key of the point being repaired, the event time, the window index, the action type and parameters, the policy confidence, the original value summary, and the version number. The default confidence is obtained by mapping the action probability output by the policy or the improvement amount of the reconstructed residual and is limited to between 0 and 1. The repair results and metadata are written atomically, and the key consists of the primary key, the event time, and the version number to achieve idempotency. When the repair action fails or the timeout exceeds 200ms, the repair is rolled back and the corresponding point is marked as an anomaly and passed down downstream. At the same time, the reason for the failure is recorded for auditing.

[0079] Step S6: Output the repaired data stream and repaired metadata for downstream applications to consume;

[0080] Optionally, the output adopts a binary or text serialization format with pattern and includes a version field. The field includes at least the primary key, event time, window index, repair flag, anomaly indicator, post-repair value, and repair metadata summary. The default partitioning strategy is based on the primary key or source identifier to ensure that the same key is ordered. When the downstream is an alarm or monitoring scenario, the anomaly confidence level and threshold snapshot summary are output simultaneously for verification.

[0081] In one embodiment, acquiring a real-time data stream includes receiving data from an IoT device, a financial transaction system, or a cloud-native application, and windowing processing includes a sliding window or a scrolling window based on event time, combined with a watermark to handle late and out-of-order data.

[0082] Specifically, input data is sorted and cached according to the source identifier, partition key, and event time in the message header. The default depth of the out-of-order buffer is 3 seconds, corresponding to a memory limit of no more than 20% of the available memory for a single parallelism. When the out-of-order data exceeds the buffer, only the incremental calculation of the window that has not yet exited is included. The water level is updated every 1 second by default and aligned with the window exit trigger, which can be adjusted within the range of 200ms to 2 seconds. When a sudden increase in the number of data at the source causes back pressure, the water level update interval is extended to 2 seconds and a window-level delayed exit strategy is enabled to ensure the minimum executable requirement for the anomaly detection and repair link.

[0083] In one embodiment, the anomaly score includes a weighted combination of contextual inconsistency based on self-attention weights and bias based on reconstruction error.

[0084] The adaptive threshold is dynamically updated based on statistics within the window to adapt to changes in data distribution;

[0085] The calculation definitions for the weighted combination of anomaly scores and the dynamic update of the adaptive threshold are as follows:

[0086] Step E1, in window number... Within the event time window, for the first For each data point, the context inconsistency and reconstruction bias are linearly synthesized after robust normalization to obtain the score:

[0087] ,

[0088] in, For the first Anomaly scores for each data point This is a normalized value for the degree of context inconsistency. To reconstruct the normalized value of the deviation, For window The weighting coefficients, Indexing data points within the window, The window number;

[0089] Normalize:

[0090] ,

[0091] in, For measurement The normalization result, when have to ,when have to , For the original measurement, For window The sample set for this metric, For the median operator, For the median absolute deviation operator, It is a scaling constant. It is the numerical stability constant;

[0092] Perform weight updates:

[0093] , , ,

[0094] in, For window The weighting coefficients, These are the reliability measures for the two types of evidence, and For window The two types of primitive metric sets;

[0095] Step E2, evaluate the rating set within the window. Calculate the robust threshold and update it over time:

[0096] ,

[0097] in, For window The threshold, The threshold of the previous window, This is the time smoothing coefficient. For set At the upper quantile level quantiles at , The quantile level parameter (the smaller the value, the more it is biased towards the high tail). This is the margin coefficient. The median absolute deviation of the rating set is used; the ratings are then mapped to anomaly masks.

[0098] ,

[0099] in, For the first Anomaly indication for each data point, For indicator functions, Rate this point. This is the current threshold.

[0100] Step E3, when the water level line causes late data to be incorporated into the window. When making up the calculation, only the affected ones Perform an incremental update, then recalculate according to step E2. With the mask, there is no need to rescan the already stable window;

[0101] Specifically, the fusion of anomalous evidence and threshold decision-making are decoupled into two layers: weighted synthesis within the window and time update across the window. The former weakens the pull of extreme values ​​by robustly normalizing the inconsistency of context and the reconstruction bias, and then allocates the contribution by adjusting the weights inversely with stability, so that evidence from more stable sources has a greater say when noise increases or the model is locally mismatched. The latter constructs the threshold using quantiles and median absolute deviation, and uses exponential smoothing to suppress short-term fluctuations, ensuring that the alarm rhythm fits the distribution changes without being overly sensitive. The mask mapping maintains a binary form, which is convenient for downstream reinforcement learning strategies to consume directly. At the same time, the incremental re-evaluation strategy is compatible with the water level mechanism, which can quickly correct the threshold and score in the case of out-of-order and late arrival, avoiding large-scale recalculation from disturbing the real-time link.

[0102] Furthermore, the context inconsistency can be obtained by inverting the attention aggregation of adjacent positions within the same window, and the reconstruction bias can be obtained by the reconstruction residual of the encoding-decoding path. Both are first subjected to robust scaling to suppress extreme values. The default scaling constant is 1, the numerical stability constant is 1e-6, the time smoothing coefficient is 0.2, the quantile level is 0.99, and the margin coefficient is 0.5, which can be adjusted within the ranges of 0.5~2, 1e-8~1e-3, 0.05~0.5, 0.95~0.999, and 0~2. When the window sample size is less than 10 or the quantile estimation is unstable, the threshold calculation reverts to using the threshold of the previous stable window and limits the single change to no more than 20% of the previous value. Late data supplementation only performs incremental updates on the affected statistics and is limited to the most recent 3 unstable windows.

[0103] In one embodiment, the input to the Transformer model includes spatial embeddings of temporal location codes and device / geographic / topological identifiers, and the encoder extracts cross-temporal and cross-device correlations through multi-head self-attention;

[0104] Similarly, the temporal location encoding is monotonically increasing within the same group and aligned with the event time. Spatial embedding is initialized using a learnable lookup table. Cold-start devices are initialized with the mean and then gradually updated during the online phase. By default, the temporal location encoding dimension accounts for 25% to 50% of the total hidden dimensions, and the spatial embedding dimension accounts for no more than 25%. Missing values ​​are masked during attention calculation. When the cross-device correlation is weaker than the threshold, it automatically degenerates into self-attention only for the temporal dimension to reduce invalid computation.

[0105] In one embodiment, anomalies are labeled with types based on anomaly scores and contextual features, and the types include at least one or more of data quality anomalies and statistical anomalies.

[0106] In one embodiment, the state of the reinforcement learning policy includes spatiotemporal features, anomaly mask, and repair history;

[0107] The action space includes interpolation, replacement, deletion, and their parameterized variants;

[0108] The reward function integrates reconstruction error, latency penalty, and business constraints to ensure consistency.

[0109] The steps for multi-objective reward aggregation and policy optimization in the reward function are as follows:

[0110] Step F1 aggregates the refactoring error, latency penalty, and business constraint consistency into an immediate reward, treating each event time window as a step, and then... Status and actions receive immediate rewards:

[0111] ,

[0112] in, For the first Instant rewards for each step To reconstruct the normalized value of the error, This is the normalized value of the delay. Aggregated penalties for violations of business constraints The weights of the three terms are given, and their sum is given. , The window number;

[0113] Perform error normalization:

[0114] ,

[0115] in, For the first Reconstruction error of step, For the recent error sample set, The median, This represents the median absolute deviation. It is the numerical stability constant;

[0116] Perform time delay normalization:

[0117] ,

[0118] in, The additional end-to-end latency caused by this repair step For business target latency threshold, For recent delay sets A robust scale, To obtain the largest operator;

[0119] Define the penalties for constraint violations:

[0120] ,

[0121] in, To constrain the number of entries, For the first The power to impose penalties under the constraints of the law For hinge operators, For the first Business constraint functions, For the penalty, For the first Step status (including spatiotemporal features, mask and repair history). This is the repair action for this step;

[0122] Define adaptive weights:

[0123] ,

[0124] in, For the first Item weight, For robust reliability measurement, For the corresponding historical sequence;

[0125] Step F2, policy update and stabilization based on policy gradient, using actor-commentator update with baseline:

[0126] ,

[0127] in, For strategy parameters, For the policy learning rate, For the probability of a conditional action, For the advantages of time-difference, The entropy coefficient, For Shannon entropy, Let be the value function, with parameters as follows: , As a discount factor, Indicates by threshold Operators for global norm clipping;

[0128] In the formula: value parameters are updated using least squares. ,in For the value learning rate, gradients are pruned using the global norm, with a threshold of [value missing]. Introducing target value network parameters Perform exponential sliding: ,in The sliding coefficient is used for sampling, and the nearest sampling is adopted. Small batches are processed sequentially within a window, with entropy terms suppressing premature convergence, and adjustments made as needed. Perform zero-mean, unit-variance standardization;

[0129] Step F3, for auditing purposes, and In the event of a drift caused by hot updates in the configuration center, the trigger logic in step E2 can be used to trigger a rollback or a gray-scale rollout.

[0130] Specifically, this approach maps the three types of objectives to comparable dimensions and then performs linear aggregation, unifying the direction to "the smaller the better," transforming the negative sign into a reward signal to avoid scale tension between multiple objectives. The error term employs robust scaling to reduce the pull of extreme values, while the time delay term constructs a one-sided hinge around the target threshold, focusing on the over-budget range. The constraint term is weighted by item and allows first- or second-order penalties, balancing sparsity and smoothness. The weight side uses the reciprocal of robust divergence as reliability, automatically allocating emphasis according to environmental noise and business rhythm, facilitating cross-scenario migration. The strategy optimization adopts an actor-critic framework, combined with entropy regularization and advantage standardization, resulting in more stable convergence. The critic uses mean squared error regression and introduces norm pruning and sliding updates of the target network to suppress estimation jitter and gradient bursts. In the context of data flow, a window is used as the time step, forming a natural connection with the aforementioned encoding, scoring, and threshold modules. Parameters can be hot-updated on demand at the configuration layer to support auditing and playback processes. This solution is flexible against online disturbances and distribution drift, while reserving sufficient margin for subsequent repair actions.

[0131] Specifically, the default weights for the three reward components are error 0.6, latency 0.2, and constraint 0.2. The default end-to-end latency threshold is 100ms, the target discount factor is 0.99, the learning rates for policy and value are 1e-4 and 5e-4, respectively, the entropy regularization coefficient is 0.01, the gradient pruning threshold is 1.0, the mini-batch size is 4 windows, and the sliding coefficient for the target value is 0.99. When the variance of the advantage estimation abnormally increases or the value loss continuously exceeds the threshold, the policy update is paused and reverted to the previous stable version. If necessary, it is downgraded to fixed rule interpolation or replacement until stability is restored.

[0132] In one embodiment, the repair execution includes repairing anomalies point by point or in segments, and generating repair metadata containing repair location, action type, parameters, and confidence level; the output supports idempotent writes and replayable recalculation; similarly, the idempotent key is composed of the source identifier, event time, window index, and version number, and the transactional commit is triggered when the window is completed and the waterline advances; replayability is achieved by recording snapshots of the input offset, threshold, weight, and adaptive parameters, and by default, the snapshot of the most recent 24 hours is retained for fault recovery; when the downstream rejects the transaction or returns a retry prompt, exponential backoff retries are adopted and a maximum of 3 retries are made. If the number of retries is exceeded, the message is transferred to the compensation channel to wait for off-peak recalculation.

[0133] In one embodiment, the online update of the Transformer model and reinforcement learning strategy is triggered based on the drift detection results. The online update adopts a canary release and retains the rollback path.

[0134] In one embodiment, the output of step S6 includes writing the repaired data stream and repair metadata into the downstream topic or operator of the stream processing framework, and providing anomaly mask and confidence level to the downstream to support alarms and analysis.

[0135] Furthermore, the output link sets alarm thresholds for end-to-end latency and throughput. By default, the P95 latency does not exceed 200ms and the throughput fluctuation does not exceed ±20%. When continuous jitter occurs, the batch size is reduced or the partition parallelism is increased first, and then the model complexity is reduced (such as turning off the top-level encoder) to maintain the established service level.

[0136] In one embodiment, interpretable information is output for the data points that have been repaired, including attention contribution, anomaly score decomposition, and the basis for selecting repair actions, for auditing and traceability purposes.

[0137] In this embodiment, the interpretable information by default includes the top five context positions with the largest contributions and their relative contributions, the proportion of anomaly scores between the two types of evidence, a comparison of candidate remediation actions and the reasons for their adoption; when the output payload exceeds the set upper limit, the top-ranked contribution items and key threshold summaries are retained first, and the original detailed information is stored in the audit storage for on-demand query.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0139] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments. For example, all the embodiments above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A data processing method for software development applications based on AI large-scale models, characterized in that, Comprise: Step S1, acquiring a real-time data stream, and windowing the real-time data stream based on event time; Step S2, encoding the windowed data using a pre-trained Transformer model to obtain spatiotemporal features representing temporal dependence and spatial correlation; Step S3, calculating an anomaly score for each data point based on the spatiotemporal features, and generating an anomaly mask according to an adaptive threshold; Step S4, inputting the anomaly mask and the spatiotemporal features into a reinforcement learning policy to generate a repair action for the anomaly point; Step S5, executing the repair action to repair the real-time data stream, obtaining a repaired data stream, and recording repair metadata; Step S6, outputting the repaired data stream and the repair metadata for downstream applications to consume. 2.The AI large model-based software development application data processing method of claim 1, wherein The acquisition of the real-time data stream includes receiving data from Internet of Things devices, financial transaction systems, or cloud-native applications, and the windowing includes sliding windows or rolling windows based on event time, and waterlines are used to handle late and out-of-order data. 3.The AI large model-based software development application data processing method of claim 1, wherein, The anomaly score includes a weighted combination of context inconsistency based on self-attention weights and deviation based on reconstruction error; The adaptive threshold is dynamically updated according to the statistics within the window; The calculation of the weighted combination of the anomaly score and the dynamic update of the adaptive threshold is defined as follows: Step E1, for the first data point within the event time window with window sequence number , the score is linearly synthesized from the robustly normalized context inconsistency degree and reconstruction bias degree: ​ ， wherein, is an anomaly score for the th data point, is a normalized value of the context inconsistency degree, is a normalized value of the reconstruction bias degree, is a weighting coefficient of the window , and is an index of the data point within the window, is a window number. Normalization is performed: ， in, For measurement The normalization result, when have to ,when have to , For the original measurement, For window The sample set for this metric, For the median operator, For the median absolute deviation operator, It is a scaling constant. It is the numerical stability constant; Weight update is performed: ， ， ， wherein, is a weighting factor for the window , are reliability measures for the two classes of evidence, respectively, and are two sets of raw measures within the window ; Step E2, score set within window Compute robust threshold and update over time: ， in, For window The threshold, The threshold of the previous window, This is the time smoothing coefficient. For set At the upper quantile level quantiles at , For quantile level parameters, This is the margin coefficient. The median absolute deviation of the rating set is used; the ratings are then mapped to anomaly masks. ， wherein, is an anomaly indication for the th data point, is an indication function, is a score for the point, is a current threshold value; Step E3, when the waterline causes late data to be incorporated into the window of the recalculation, only the affected is incrementally updated, and the recalculation with the mask is repeated, without a need to rescan the stable window. 4.The AI large model-based software development application data processing method of claim 1, wherein, The input of the Transformer model includes time position encoding and spatial embedding of device / geography / topology identification, and the encoder extracts cross-time and cross-device correlations through multi-head self-attention. 5.The AI large model-based software development application data processing method of claim 1, wherein Based on the anomaly score and the context features, the anomaly points are labeled by type, including one or more of data quality anomaly and statistical anomaly. 6.The AI large model-based software development application data processing method of claim 1, wherein The state of the reinforcement learning policy includes the spatiotemporal features, the anomaly mask, and the repair history; The action space includes interpolation, replacement, deletion, and their parameterized variants; The reward function integrates reconstruction error, latency penalty, and business constraint consistency; The multi-objective reward aggregation in the reward function and the policy optimization steps are as follows: Step F1, aggregate the reconstruction error, the latency penalty and the service constraint consistency into an immediate reward, treat each episode time window as a step, and the window number is the state and action get the immediate reward: ， wherein, is the first instant reward of the step, is the normalized value of the reconstruction error, is the normalized value of the latency, is the aggregated penalty of the service constraint violations, is the weight of the three terms and and is , is the window number; Error normalization is performed: ， wherein, is the first reconstruction error of the step, is a set of recent error samples, is the median, is the median absolute deviation, is a numerical stability constant; Latency normalization is performed: ， in, The additional end-to-end latency caused by this repair step For business target latency threshold, For recent delay sets A robust scale, To obtain the largest operator; Define constraint violation penalty: ， wherein, is the number of constraints, is the number of constraints, is the penalty weight of the constraint, is the hinge operator, is the number of constraints, is the service constraint function, is the penalty, is the number of constraints, is the number of steps, is the repair action of the step. Define weight adaptation: ， wherein, is the first item weight, is the robust reliability measure, is the corresponding historical sequence; Step F2, policy update and stabilization based on policy gradient, using actor-critic update with baseline: ， wherein, is a policy parameter, is a policy learning rate, is a conditional action probability, is a time-difference advantage, is an entropy coefficient, is a Shannon entropy, is a value function, parameterized by , is a discount factor, denotes an operator that performs global norm clipping by a threshold ; where the value parameter is updated by least square where is the value learning rate, the gradient is clipped by global norm with threshold , the target value network parameter is introduced is exponentially smoothed: where is the smoothing coefficient, the sampling is done in time-ordered mini-batches of the most recent windows, and the entropy term suppresses overfitting. Step F3, for audit convenience, With With configuration center hot update, when drift occurs, rollback or gray scale according to the trigger logic of step E2. 7.The AI large model-based software development application data processing method of claim 1, wherein Repair execution includes point-by-point or segment-by-segment repair of anomaly points, and generates repair metadata including repair location, action type, parameters, and confidence; the output supports idempotent writing and replayable reprocessing. 8.The AI large model-based software development application data processing method of claim 1, wherein Based on the drift detection result, trigger online update of the Transformer model and the reinforcement learning policy, which adopts gray release and retains rollback path. 9.The AI large model-based software development application data processing method of claim 1, wherein, The output of step S6 includes writing the repaired data stream and the repair metadata to the downstream topic or operator of the stream processing framework, and providing anomaly mask and confidence to the downstream. 10.The AI large model-based software development application data processing method of claim 1, wherein For the repaired data points, output interpretable information including attention contribution, anomaly score decomposition, and repair action selection basis.

Images