One-dimensional discrete sequence anomaly detection and repair method and system

Abstract

The invention discloses a one-dimensional discrete sequence anomaly detection and repair method and system, and belongs to the technical field of data processing, and the method comprises the steps: carrying out the preprocessing of a one-dimensional discrete sequence, and generating an input sequence; performing forward propagation on the input sequence by using a mask language model to obtain category probability distribution of each position; calculating an anomaly score based on the probability distribution; comparing the abnormal score with a preset threshold value, and generating an abnormal position mask; and repairing the abnormal position, and using the maximum probability category in the probability distribution as a repairing value. According to the method, anomaly detection and repair are unified into a mask language model task, so that integrated end-to-end processing of detection and repair is realized, a threshold value does not need to be manually set, a repair result conforms to a global semantic mode, and the accuracy and the automation degree of anomaly processing are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a method and system for anomaly detection and repair in a one-dimensional discrete sequence. Background Technology

[0002] In fields such as intelligent connected vehicles, IoT device status monitoring, and industrial sensor data analysis, the accuracy and completeness of one-dimensional discrete sequence data (such as road classification labels, equipment status codes, and traffic flow categories) have a decisive impact on system performance. This type of sequence data is typically represented in integer encoding, such as road types (0 = unidentified, 1-7 = different road types). During data acquisition, transmission, and storage, it is susceptible to network fluctuations, equipment failures, or system errors, leading to data anomalies (such as erroneous values, missing values, or abrupt changes).

[0003] Currently, the main methods used for anomaly detection and repair techniques for one-dimensional discrete sequences are as follows: Anomaly detection methods based on reconstruction errors: These methods first reconstruct the input sequence using models such as autoencoders and LSTMs, and then determine the location of anomalies by calculating the reconstruction error (e.g., mean squared error, absolute error). However, this method requires setting an additional threshold to distinguish between normal and abnormal data, and the threshold setting is highly dependent on human experience, making it difficult to adapt to different data distribution scenarios. Furthermore, the reconstructed values ​​can only be used for anomaly detection and cannot directly provide repair values, requiring the design of additional repair strategies, thus separating the detection and repair processes.

[0004] Rule-based anomaly detection and repair methods: These methods predefine anomaly rules (e.g., "Road type 7 should not appear after road type 1"). When a sequence that violates the rules is detected, it is repaired using simple interpolation (e.g., linear interpolation, forward padding) or fixed rules. However, these methods lack the ability to perceive the semantic context of the sequence, and the repair results often do not conform to the actual business logic, destroy the pattern consistency of the sequence, and are difficult to adapt to complex and ever-changing business scenarios.

[0005] Anomaly detection methods based on classification models: Some studies treat anomaly detection as a classification task, training a classifier to distinguish between normal and abnormal samples. However, this method usually requires a large amount of labeled data, while in real-world applications, abnormal samples are scarce and labeling is costly. Furthermore, the classification results only provide a binary judgment of whether an anomaly exists, and cannot provide specific correction values, requiring additional steps to complete the correction.

[0006] The existing technologies generally have the following defects: (1) The anomaly detection and repair process is separated, requiring multiple model calculations and post-processing steps, which increases the complexity of the system; (2) The anomaly threshold depends on manual setting and lacks adaptive capability, resulting in repeated parameter tuning in different scenarios; (3) The repair process lacks global semantic awareness of the sequence context, and the repair results do not match the business logic; (4) Most methods require a large amount of labeled data, making it difficult to apply in unlabeled data scenarios.

[0007] Therefore, there is an urgent need for a one-dimensional discrete sequence anomaly handling method that can integrate anomaly detection and repair, eliminates the need for manual threshold setting, possesses contextual semantic awareness, and is applicable to unlabeled data, in order to improve data quality and reduce system maintenance costs. Summary of the Invention

[0008] This application provides a one-dimensional discrete sequence anomaly detection and repair method and system based on a mask language model, which can solve the technical problems existing in the prior art, such as the separation of anomaly detection and repair processes, the reliance on manual setting of anomaly thresholds, and the lack of contextual semantic awareness in repair results.

[0009] In a first aspect, embodiments of this application provide a method for anomaly detection and repair of one-dimensional discrete sequences, characterized in that the method includes: Preprocess one-dimensional discrete sequence data to generate the input sequence; The input sequence is forward-propagated using a masked language model to obtain the class probability distribution at each position; The anomaly score for each location is calculated based on the category probability distribution; The abnormal score is compared with a preset threshold to generate an abnormal location mask; The positions of the abnormal location mask markers are repaired using the highest probability category in the probability distribution as the repair value.

[0010] In conjunction with the first aspect, in one implementation, the preprocessing includes: mapping the original discrete values ​​to integer indices; introducing a mask special token; and randomly selecting a portion of the positions for masking processing.

[0011] In conjunction with the first aspect, in one implementation, the mask language model includes an encoder based on the Transformer architecture.

[0012] In conjunction with the first aspect, in one implementation, calculating the anomaly score includes: determining the maximum probability value in the probability distribution of each location; and generating an anomaly score based on the maximum probability value.

[0013] In conjunction with the first aspect, in one implementation, the preset threshold is automatically set based on the statistical characteristics of the verification dataset.

[0014] In conjunction with the first aspect, in one embodiment, the method further includes performing data augmentation on the input sequence during the training phase.

[0015] In conjunction with the first aspect, in one implementation, the mapping in the preprocessing is a vocabulary-based mapping that includes all possible discrete values.

[0016] Secondly, embodiments of this application provide a one-dimensional discrete sequence anomaly detection and repair system, the system comprising: The data preprocessing module is used to preprocess one-dimensional discrete sequence data to generate input sequences; The model processing module is used to perform forward propagation on the input sequence using a masked language model to obtain the class probability distribution at each position; An anomaly score calculation module is used to calculate the anomaly score for each location based on the category probability distribution; An abnormal location generation module is used to compare the abnormal score with a preset threshold and generate an abnormal location mask; The repair module is used to repair the position of the abnormal position mask mark, using the highest probability category in the probability distribution as the repair value.

[0017] In conjunction with the second aspect, in one implementation, the preprocessing includes: mapping the original discrete values ​​to integer indices; introducing a mask special token; and randomly selecting a portion of the positions for masking processing.

[0018] In conjunction with the second aspect, in one implementation, calculating the anomaly score includes: determining the maximum probability value in the probability distribution of each location; and generating an anomaly score based on the maximum probability value.

[0019] The beneficial effects of the technical solutions provided in this application include: By unifying anomaly detection and repair into a mask prediction task of a mask language model, the model can output the anomaly location mask and repair value simultaneously in a single forward propagation, achieving integrated end-to-end processing of detection and repair.

[0020] The system automatically sets anomaly thresholds based on the statistical characteristics of the validation dataset, eliminating the need for manual adjustments and enabling automatic threshold calibration.

[0021] Repair is performed using sequence context probability distribution, ensuring that the repaired values ​​conform to the global semantic pattern and guaranteeing the consistency of the repair results with business logic. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an embodiment of the one-dimensional discrete sequence anomaly detection and repair method of this application; Figure 2 This is a schematic diagram of the functional modules of an embodiment of the one-dimensional discrete sequence anomaly detection and repair system of this application; Figure 3 This is a schematic diagram of the training loss curve in a specific embodiment of this application; Figure 4 This is a schematic diagram of the training accuracy curve in a specific embodiment of this application; Figure 5 This is a schematic diagram of the distribution of anomaly scores and threshold selection on the validation set in a specific embodiment of this application; Figure 6 This is a schematic diagram of a detailed performance evaluation report of the model on the test set in a specific embodiment of this application; Figure 7 The image shows the code and screenshots illustrating the threshold selection analysis in a specific embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0025] In the first aspect, embodiments of this application provide a method for detecting and repairing anomalies in one-dimensional discrete sequences. This method can be directly applied to technical fields that require automated fault diagnosis and repair of one-dimensional discrete sequences, such as sensor data cleaning for intelligent connected vehicles, sequence error correction for Internet of Things (IoT) device status monitoring, and instruction stream verification for industrial control systems.

[0026] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the one-dimensional discrete sequence anomaly detection and repair method of this application. Figure 1 As shown, the one-dimensional discrete sequence anomaly detection and repair method includes: Step S1: Preprocess the one-dimensional discrete sequence data to generate the input sequence.

[0027] Step S2: Use the masked language model to perform forward propagation on the input sequence to obtain the class probability distribution at each position.

[0028] Step S3: Calculate the anomaly score for each location based on the category probability distribution.

[0029] Step S4: Compare the abnormal score with a preset threshold to generate an abnormal location mask.

[0030] Step S5: Repair the position of the abnormal location mask mark, using the highest probability category in the probability distribution as the repair value.

[0031] In this embodiment, by unifying anomaly detection and repair into a mask prediction task of a mask language model, the model can output the anomaly location mask and repair value simultaneously in a single forward propagation. This avoids the separate process of detection followed by repair in traditional technologies, significantly reducing system complexity and improving processing efficiency, and achieving integrated end-to-end processing of detection and repair.

[0032] The system automatically sets anomaly thresholds based on the statistical characteristics of the validation dataset (such as the 98th percentile), eliminating the need for manual adjustments based on experience. This solves the problem of poor scenario adaptability caused by the reliance on expert experience in threshold setting in existing technologies, and enables automatic threshold calibration.

[0033] Repair is performed by using the probability distribution of sequence context (by calculating the anomaly score and selecting the category with the highest probability through S = 1 - max(P)). This ensures that the repaired value conforms to the global semantic pattern (such as the business rules of road type sequences), avoiding the sequence pattern destruction caused by simple interpolation in traditional methods. This significantly improves the repair accuracy and ensures the consistency of the business logic of the repair results.

[0034] Furthermore, in one embodiment, the preprocessing includes: mapping the original discrete values ​​to integer indices; introducing a mask special token; and randomly selecting a subset of positions for masking.

[0035] In this embodiment, a vocabulary is constructed by converting each possible value in the original discrete sequence (such as road type 1-7, equipment status code, etc.) into a unique integer index. This operation transforms non-numerical categories into a numerical form that the model can process, making the sequence data mathematically manipulable.

[0036] Special markers are added to the vocabulary to represent the positions to be predicted. When a position in the sequence is marked with this special marker, the model needs to predict the original discrete value of that position, thereby establishing a prediction relationship between the context and the target value.

[0037] During training, a portion of the sequence is randomly selected at a fixed percentage (e.g., 15%) and replaced with special labels. The masked positions serve as prediction targets, while the remaining positions act as contextual inputs, forming self-supervised learning samples. This mechanism enables the model to learn the global contextual dependencies of the sequence, rather than simply memorizing fixed patterns.

[0038] In summary, through the random masking mechanism, the model can learn the normal distribution pattern of sequences without relying on manually labeled abnormal data. For example, in the road grade data restoration implementation case, the model only needs 60 rounds of training to converge (without labeling abnormal samples), significantly reducing data preparation costs and improving training efficiency.

[0039] The random masking strategy enables the model to deeply understand the semantic context of the sequence. For example, in a road type sequence, the model can recognize the business rule that "a highway is usually followed by a highway exit," so that the corrected value is directly derived from the maximum value of the probability distribution. For instance, in an implementation case, the model automatically corrects the error value 11 to 1 (highway) which is consistent with the context, instead of simply interpolating to 6 (ordinary road), which greatly improves the accuracy of the correction.

[0040] Preprocessing standardizes the original discrete sequences into a unified input format, eliminating process fragmentation caused by differences in data formats. Implementation examples show that this step simplifies the end-to-end process of road grade data restoration from "manual rules + interpolation" to a single model processing of "input-prediction-output," significantly reducing system complexity.

[0041] Furthermore, in one embodiment, the mask language model described above includes an encoder based on the Transformer architecture.

[0042] In this embodiment, an encoder based on the Transformer architecture is used, which enables the model to effectively capture long-range dependencies in the sequence, thereby improving the accuracy of anomaly detection and ensuring that the repair results conform to the global context semantics, avoiding the repair deviation caused by local feature dependencies in traditional methods.

[0043] Furthermore, in one embodiment, calculating the anomaly score includes: determining the maximum probability value in the probability distribution of each location; and generating an anomaly score based on the maximum probability value.

[0044] In this embodiment, the anomaly score calculation method is based on the probability distribution characteristics output by the mask language model, and constructs anomaly measurement by directly utilizing the maximum probability value in the probability distribution.

[0045] In one specific embodiment, for the probability distribution vector of each position in the sequence, its maximum probability value is first determined, and then 1 minus this value is calculated as the anomaly score. This design naturally gives the anomaly score the characteristic of mathematical normalization, directly reflecting the model's confidence in the current position prediction. When the model's prediction of a certain position is highly certain (maximum probability value close to 1), the anomaly score is close to 0, indicating that the position is normal. Conversely, when the prediction uncertainty is high (maximum probability value close to 0), the anomaly score is close to 1, indicating that the position is abnormal.

[0046] By strictly limiting the score values ​​to the range of 0 to 1, there is no need to adjust the scale for different datasets, making the anomaly detection results stable and comparable in different scenarios.

[0047] Anomaly scores directly quantify the uncertainty of model predictions, making the detection results clearly interpretable and avoiding the ambiguity of judgment caused by indirect indicators such as reconstruction errors in traditional methods.

[0048] The normalization properties of the scores allow the threshold to be automatically determined based on the statistical properties of the validation dataset (such as the 98th percentile), without the need for manual adjustment based on experience, which significantly reduces the complexity of system deployment.

[0049] The anomaly score and the repair value (based on the highest probability category) originate from the same probability distribution, ensuring that the detection and repair decisions are semantically consistent and avoiding the repair bias caused by the disconnect between detection and repair logic in traditional methods.

[0050] Furthermore, in one embodiment, the aforementioned preset threshold is automatically set based on the statistical characteristics of the verification dataset.

[0051] In this embodiment, the automatic setting method for the preset threshold is based on the statistical characteristics of the abnormal score distribution of the verification dataset.

[0052] In one specific embodiment, after model training is complete, the abnormal scores of all samples are calculated using the validation dataset, the score distribution characteristics are statistically analyzed, and a specific quantile (such as the 98th percentile) is selected as a threshold. This threshold is determined through a data-driven approach, ensuring that samples with abnormal scores exceeding the threshold are classified as abnormal, without relying on manual experience for setting. This mechanism directly utilizes the statistical properties of the data, ensuring a high degree of match between the threshold and the actual data distribution.

[0053] The threshold is automatically determined by the validation dataset, avoiding the problem of poor scenario adaptability caused by manually setting the threshold in traditional methods, and significantly reducing the complexity of system deployment.

[0054] The threshold is generated based on the statistical characteristics of actual data, enabling the system to automatically adapt to different business scenarios and changes in data distribution (such as differences in road grade sequences in different regions) without the need to readjust parameters for each scenario.

[0055] Automated threshold setting simplifies the process from training to application to three steps: training, threshold generation, and deployment, significantly shortening the deployment cycle.

[0056] Furthermore, in one embodiment, the above method further includes data augmentation processing of the input sequence during the training phase.

[0057] In this embodiment, data augmentation is used to transform the input sequence during the model training phase, generating diverse training samples by randomizing sequence fragments, adding noise, or applying sequence transformations.

[0058] In one specific embodiment, the input sequence is randomly pruned, its positions perturbed, or its values ​​replaced in each training iteration to simulate the data variability in real-world applications. This approach does not alter the discrete nature of the sequence; it only enhances the model's ability to learn contextual dependencies through data-level expansion.

[0059] By generating diverse training samples, the model can adapt to the varying sequence distributions across different business scenarios. In the road-level data restoration case, the enhanced model improved anomaly detection accuracy in unseen areas, avoiding the poor scenario adaptability issues caused by the limited training data in traditional methods.

[0060] Data augmentation makes the model more tolerant of input noise and outliers, and enables the model to converge in fewer iterations while maintaining higher detection accuracy, thus achieving a balance between training efficiency and performance.

[0061] Furthermore, in one embodiment, the mapping in the preprocessing described above is a vocabulary-based mapping, which includes all possible discrete values.

[0062] In this embodiment, the vocabulary mapping mechanism achieves a standardized conversion from raw sequence data to integer indices by constructing a complete vocabulary that covers all possible discrete values.

[0063] In one specific embodiment, during the preprocessing stage, the system analyzes all discrete values ​​in the training dataset (such as road types 1-7, equipment status codes, etc.), assigns a unique integer index to each unique value, and forms a vocabulary containing all possible values. This vocabulary serves as the mapping basis, ensuring that any legal discrete value in the sequence can be accurately converted into a numerical form that the model can process, avoiding mapping errors caused by unknown values, and providing a unified input format for subsequent masking processing.

[0064] The vocabulary covers all possible discrete values, so that the model does not need to handle unknown values ​​when processing new data, avoiding system interruptions or misjudgments caused by missing values ​​in traditional methods, and significantly improving system reliability.

[0065] A complete vocabulary enables the model to adapt to sequence expansions in different business scenarios (such as adding new road types or equipment status codes). In the case of road level data repair, the system's accuracy in detecting anomalies in new area data is improved, without the need to retrain the model.

[0066] The vocabulary construction is automated, eliminating the need for manual mapping and simplifying the preprocessing steps from manually configuring mapping rules to automatically building the vocabulary, thus shortening system deployment time.

[0067] A unified vocabulary ensures consistent mapping logic across different sequences, avoids semantic breaks in context caused by mapping differences, and ensures that the repaired values ​​are highly consistent with the overall pattern of the sequence, thereby improving the business rationality of the repair results.

[0068] Secondly, this application also provides a one-dimensional discrete sequence anomaly detection and repair system. This system can be directly applied to technical fields that require automated fault diagnosis and repair of one-dimensional discrete sequences, such as sensor data cleaning for intelligent connected vehicles, sequence error correction for Internet of Things (IoT) device status monitoring, and instruction stream verification for industrial control systems.

[0069] In one embodiment, reference is made to Figure 2 , Figure 2 This is a functional module diagram of an embodiment of the one-dimensional discrete sequence anomaly detection and repair system of this application. Figure 2 As shown, the one-dimensional discrete sequence anomaly detection and repair system includes: Data preprocessing module 1 is used to preprocess one-dimensional discrete sequence data to generate input sequences.

[0070] Model processing module 2 is used to perform forward propagation on the above input sequence using a masked language model to obtain the class probability distribution at each position.

[0071] Anomaly score calculation module 3 is used to calculate the anomaly score for each position based on the above category probability distribution.

[0072] The abnormal location generation module 4 is used to compare the above-mentioned abnormal scores with a preset threshold to generate an abnormal location mask.

[0073] Repair module 5 is used to repair the position of the above-mentioned abnormal position mask mark, using the highest probability category in the above probability distribution as the repair value.

[0074] In this embodiment, by unifying anomaly detection and repair into a mask prediction task of a mask language model, the model can output the anomaly location mask and repair value simultaneously in a single forward propagation. This avoids the separate process of detection followed by repair in traditional technologies, significantly reducing system complexity and improving processing efficiency, and achieving integrated end-to-end processing of detection and repair.

[0075] The system automatically sets anomaly thresholds based on the statistical characteristics of the validation dataset (such as the 98th percentile), eliminating the need for manual adjustments based on experience. This solves the problem of poor scenario adaptability caused by the reliance on expert experience in threshold setting in existing technologies, and enables automatic threshold calibration.

[0076] Repair is performed by using the probability distribution of sequence context (by calculating the anomaly score and selecting the category with the highest probability through S = 1 - max(P)). This ensures that the repaired value conforms to the global semantic pattern (such as the business rules of road type sequences), avoiding the sequence pattern destruction caused by simple interpolation in traditional methods. This significantly improves the repair accuracy and ensures the consistency of the business logic of the repair results.

[0077] The functions of each module in the above-mentioned one-dimensional discrete sequence anomaly detection and repair system correspond to the steps in the above-mentioned one-dimensional discrete sequence anomaly detection and repair method embodiment, and their functions and implementation processes will not be described in detail here.

[0078] In one specific embodiment, the integrated inference process includes anomaly detection and value repair. Anomaly detection involves inputting the sequence to be detected into a trained model to obtain the probability distribution at each position. Anomaly score S_i = 1 - max(P_i) is calculated, where P_i is the probability distribution vector at the i-th position. A higher S_i value indicates a lower confidence level in the model's prediction for that position, and a greater likelihood that the position is an anomaly. An anomaly mask is generated by comparing S_i with a preset threshold (which can be automatically calibrated). Value repair involves replacing the value of positions identified as anomalies with the model's predicted argmax category (i.e., the category with the highest probability). Non-anomaly positions retain their original values.

[0079] In another specific embodiment, a one-dimensional discrete sequence anomaly detection and repair method based on a mask language model is provided for application in vehicle road type data repair. This embodiment addresses the identification errors and unidentified issues in vehicle road type sequence data caused by outdated map data and network problems, aiming to improve the accuracy of fuel consumption analysis.

[0080] In intelligent connected vehicle systems, road type sequence data (including 0 = unidentified, 1-7 = different road types) has a crucial impact on fuel consumption analysis. Due to outdated map data and network transmission issues, the road type sequence per minute for vehicles often contains erroneous values ​​(such as 11), severely affecting the accuracy of fuel consumption analysis. This embodiment effectively solves the above problems through a one-dimensional discrete sequence anomaly detection and repair method based on a masked language model.

[0081] First, prepare the training data, using historical road type sequence data as the training set, as shown in the example below: [4, 4,4, ..., 4, 3, 4, 4, ..., 4, 5, 5, ..., 5, 2, 5, ..., 5, 3, 3, ..., 3, 7, 7,..., 7, 3, 3, 2, 5, 2, ..., 2]. The sequence contains discrete values ​​of road type 0-7, where 4 represents ordinary roads, 5 represents provincial and national highways, 6 represents urban expressways, 7 represents highways, and 3 represents other types.

[0082] The training data is then preprocessed, including mapping the original discrete values ​​to integer indices, constructing a vocabulary that includes all possible discrete values ​​(0-7), introducing a mask token [MASK], and randomly selecting 15% of the positions for masking.

[0083] Next, the model was built and trained. The TinyTransformer model was used, with parameters set to d_model=64, nhead=4, and nlayers=4. The daily data length was set to T=128, and the sliding step size was T / 2. Historical data was used for training, which lasted for 60 epochs. The loss function continued to decrease until convergence.

[0084] Finally, the sequence of road types to be detected [4, 4, 4, ..., 4, 4, 11, 4, ..., 4, 5, 5, ..., 5, 5, 5, 5, ..., 5, 3, 3, ..., 3, 7, 7, ..., 7, 3, 3, 2, 5, 2, ..., 2] is used as the input sequence, which contains error values ​​(outliers at positions 111, 159, 213, and 259 in the example). The processing flow includes inputting the input sequence into a trained mask language model. The model generates a class probability distribution for each position. Anomaly scores S_i = 1 - max(P_i) are calculated, where P_i is the probability distribution vector at the i-th position. A threshold is automatically set based on the statistical characteristics (98th percentile) of the validation dataset. The anomaly scores are compared with the threshold to generate anomaly location masks. For anomaly locations, the original value is replaced with the class with the highest probability in the probability distribution. Original values ​​are retained for non-abnormal locations.

[0085] Output anomaly location mask: [0, 0, 0, ..., 0, 0, 1, 0, ..., 0, 0, 0, ..., 0, 0, 0, 0, ..., 0, 0, 0, ..., 0, 0, 0, ..., 0, 0, 0, 0, 0, ..., 0, 1, 0, ..., 0, 0, 0, ..., 0, 0, 0, 0, 0, ..., 0, 1, 0, ..., 0, 0, 0, ..., 0, 0, 0, 0, 0, ..., 0, 1, 0, ..., 0, 0, 0, ..., 0, 0, 0, 0, 0, ..., 0, 1, 0, ..., 0], where a value of 1 in the mask indicates that an anomaly was detected.

[0086] The output corrected sequence is: [4, 4, 4, ..., 4, 4, 4, 4, ..., 4, 5, 5, ..., 5, 5, 5, 5, ..., 5, 3, 3, ..., 3, 7, 7, ..., 7, 3, 3, 2, 2, 2, ..., 2], where all abnormal positions are corrected to the correct values ​​that conform to the context semantics, such as 11 being corrected to 4, and 55 in the original sequence being corrected to 5, etc.

[0087] like Figure 3 The diagram shown is a schematic of the training loss curve. This method is implemented using a neural network. The training evaluation criterion is to reduce the loss (the difference between the predicted value and the actual value). The loss curve indicates how long the training process will take to converge. Once convergence is achieved, further training is no longer required.

[0088] like Figure 4 As shown, this is a schematic diagram of the training accuracy curve. This method is implemented using a neural network approach. The training uses improving accuracy (the predicted value and the actual value are consistent) as the evaluation criterion. The accuracy curve indicates how long the training process will take to converge. Once convergence is achieved, further training is no longer needed.

[0089] like Figure 5 As shown, the diagram illustrates the distribution of outlier scores on the validation set and the threshold selection, along with the curve showing the relationship between threshold selection and algorithm accuracy, used to select the optimal threshold.

[0090] like Figure 6 As shown, this is a schematic diagram of the detailed performance evaluation report of the model on the test set, with the loss and accuracy curves of the algorithm on the test data, reflecting the actual effect of the algorithm.

[0091] like Figure 7 As shown, the threshold selection analysis code and screenshot diagram are presented. The code screenshot is used as evidence of patent implementation. This embodiment effectively identifies erroneous values ​​in road type sequences in practical applications. The repaired sequence conforms to the business logic of road types (e.g., highways are usually followed by highway exits), avoiding semantic breaks caused by simple interpolation. It achieves integrated detection and repair processing, completing the entire process in a single forward propagation, significantly improving system processing efficiency. Thresholds are automatically set, requiring no manual adjustment, greatly reducing system maintenance costs.

[0092] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0093] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0094] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0095] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0096] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0098] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for anomaly detection and repair in a one-dimensional discrete sequence, characterized in that, The method includes: Preprocess one-dimensional discrete sequence data to generate the input sequence; The input sequence is forward-propagated using a masked language model to obtain the class probability distribution at each position; The anomaly score for each location is calculated based on the category probability distribution; The abnormal score is compared with a preset threshold to generate an abnormal location mask; The positions of the abnormal location mask markers are repaired using the highest probability category in the probability distribution as the repair value.

2. The one-dimensional discrete sequence anomaly detection and repair method as described in claim 1, characterized in that, The preprocessing includes: mapping the original discrete values ​​to integer indices; introducing a mask special token; and randomly selecting some positions for masking.

3. The one-dimensional discrete sequence anomaly detection and repair method as described in claim 1, characterized in that, The masking language model includes an encoder based on the Transformer architecture.

4. The one-dimensional discrete sequence anomaly detection and repair method as described in claim 1, characterized in that, The calculation of the anomaly score includes: determining the maximum probability value in the probability distribution of each location; and generating an anomaly score based on the maximum probability value.

5. The one-dimensional discrete sequence anomaly detection and repair method as described in claim 1, characterized in that, The preset threshold is automatically set based on the statistical characteristics of the validation dataset.

6. The one-dimensional discrete sequence anomaly detection and repair method as described in claim 1, characterized in that, The method also includes performing data augmentation on the input sequence during the training phase.

7. The one-dimensional discrete sequence anomaly detection and repair method as described in claim 2, characterized in that, The mapping in the preprocessing is a vocabulary-based mapping, which includes all possible discrete values.

8. A one-dimensional discrete sequence anomaly detection and repair system, characterized in that, The system includes: The data preprocessing module is used to preprocess one-dimensional discrete sequence data to generate input sequences; The model processing module is used to perform forward propagation on the input sequence using a masked language model to obtain the class probability distribution at each position; An anomaly score calculation module is used to calculate the anomaly score for each location based on the category probability distribution; An abnormal location generation module is used to compare the abnormal score with a preset threshold and generate an abnormal location mask; The repair module is used to repair the position of the abnormal position mask mark, using the highest probability category in the probability distribution as the repair value.

9. The one-dimensional discrete sequence anomaly detection and repair system as described in claim 8, characterized in that, The preprocessing includes: mapping the original discrete values ​​to integer indices; introducing a mask special token; and randomly selecting some positions for masking.

10. The one-dimensional discrete sequence anomaly detection and repair system as described in claim 8, characterized in that, The calculation of the anomaly score includes: determining the maximum probability value in the probability distribution of each location; and generating an anomaly score based on the maximum probability value.

Images