Abnormal data cleaning method and device for database and electronic equipment

Abstract

This application relates to the field of abnormal data cleaning, and specifically discloses an abnormal data cleaning method, apparatus and electronic device for databases. It first acquires information data from the front-end processing terminal and historical abnormal data cleaning data, then uses deep learning technology to extract features and perform correlation analysis on the two, and finally uses a classifier to obtain abnormal data cleaning processing labels, so as to effectively improve the efficiency and quality of data cleaning, thereby improving the detection speed of abnormal data and reducing the need for manual intervention.

Description

Technical Field

[0001] This application relates to the field of abnormal data cleaning, and more specifically, to an abnormal data cleaning method, apparatus, and electronic device for databases. Background Technology

[0002] With the advancement of big data technology, terminals processing information data need to analyze data transmitted from front-end terminals (such as control centers or station operator workstations) to obtain valid analytical results. However, the data transmitted by front-end terminals does not always meet processing standards and often contains various anomalies. Directly analyzing this abnormal data can lead to erroneous results. To ensure the accuracy of data analysis, anomaly detection and processing of the data transmitted by front-end terminals must be performed first. While current technology can detect anomalies item by item, it is not efficient for processing large amounts of information data and cannot quickly and effectively identify and eliminate these anomalies.

[0003] Therefore, there is a need for a method, apparatus, and electronic device for cleaning abnormal data in databases. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a method, apparatus, and electronic device for cleaning abnormal data in a database. It first acquires information data from a front-end processing terminal and historical abnormal data cleaning data. Then, it utilizes deep learning technology to extract features and perform correlation analysis on both. Finally, it uses a classifier to obtain abnormal data cleaning processing labels, effectively improving the efficiency and quality of data cleaning, thereby increasing the detection speed of abnormal data and reducing the need for manual intervention.

[0005] According to one aspect of this application, a method for cleaning abnormal data in a database is provided, comprising:

[0006] Acquire information data from the front-end processing terminal and clean historical abnormal data;

[0007] Extract terminal information data segment granular text understanding feature vectors and historical abnormal data cleaning text global feature vectors from the information data of the front-end processing terminal and the historical abnormal data cleaning data.

[0008] Based on the granular text understanding feature vector of the terminal information data segment and the global feature vector of the historical abnormal data cleaning text, an abnormal data cleaning processing label is obtained.

[0009] According to another aspect of this application, an anomalous data cleaning apparatus for a database is provided, comprising:

[0010] The database data information acquisition module is used to acquire information data from the front-end processing terminal and historical abnormal data cleaning data;

[0011] The database data information extraction module is used to extract terminal information data segment granular text understanding feature vectors and historical abnormal data cleaning text global feature vectors from the information data of the front-end processing terminal and the information data of the historical abnormal data cleaning.

[0012] The abnormal data cleaning and processing label representation module is used to obtain abnormal data cleaning and processing labels based on the granular text understanding feature vector of the terminal information data segment and the global feature vector of the historical abnormal data cleaning text.

[0013] According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing computer program instructions that, when executed by the processor, cause the processor to perform the abnormal data cleaning method for a database as described above.

[0014] Compared with existing technologies, this application provides an abnormal data cleaning method, apparatus, and electronic device for databases. It first acquires information data from the front-end processing terminal and historical abnormal data cleaning data, then uses deep learning technology to extract features and perform correlation analysis on the two, and finally uses a classifier to obtain abnormal data cleaning processing labels, thereby effectively improving the efficiency and quality of data cleaning, thereby increasing the detection speed of abnormal data and reducing the need for manual intervention. Attached Figure Description

[0015] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 This is a flowchart of an abnormal data cleaning method for a database according to an embodiment of this application.

[0017] Figure 2 This is a flowchart illustrating the process of extracting features from the information data of the front-end processing terminal to obtain a terminal information data segment granular text understanding feature vector in the abnormal data cleaning method for databases according to an embodiment of this application.

[0018] Figure 3This is a flowchart illustrating the process of preprocessing information data from the front-end processing terminal to obtain multiple terminal information data text understanding feature vectors in an abnormal data cleaning method for a database according to an embodiment of this application.

[0019] Figure 4 This is a flowchart illustrating the process of performing feature encoding on the text understanding feature vectors of the multiple terminal information data to obtain the granular text understanding feature vectors of the terminal information data segment in the abnormal data cleaning method for databases according to embodiments of this application.

[0020] Figure 5 This is a block diagram of an abnormal data cleaning apparatus for a database according to an embodiment of this application.

[0021] Figure 6 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0022] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0023] Figure 1 This is a flowchart of an abnormal data cleaning method for a database according to an embodiment of this application. Figure 1 As shown, the abnormal data cleaning method for a database according to an embodiment of this application includes: S110, acquiring information data of a front-end processing terminal and historical abnormal data cleaning data; S120, extracting terminal information data segment granular text understanding feature vectors and historical abnormal data cleaning text global feature vectors from the information data of the front-end processing terminal and the historical abnormal data cleaning data; S130, obtaining abnormal data cleaning processing labels based on the terminal information data segment granular text understanding feature vectors and the historical abnormal data cleaning text global feature vectors.

[0024] In the aforementioned abnormal data cleaning method 100 for databases, step S110 involves acquiring information data from the front-end processing terminal and historical abnormal data cleaning data. It should be understood that with the advancement of big data technology, terminal devices need to analyze data transmitted from the front-end when processing information data to obtain effective analysis results. However, the data transmitted from the front-end does not always meet processing standards and often contains various anomalies. Directly analyzing this abnormal data may lead to erroneous analysis results. To ensure the accuracy of data analysis, anomaly detection and processing of the data transmitted from the front-end must be performed first. Although current technology can detect anomalies one by one, it is not efficient for processing large amounts of information data, making it difficult to quickly and effectively identify and remove these anomalies. Therefore, in the technical solution of this application, by acquiring information data from the front-end processing terminal and historical abnormal data cleaning data, and combining it with deep learning technology, anomaly data cleaning processing labels are obtained. Here, by effectively improving the efficiency and quality of data cleaning, the detection speed of abnormal data can be accelerated, while reducing reliance on manual intervention.

[0025] Specifically, the information data from the front-end processing terminal provides a real-time and instantaneous data stream, reflecting the current system operating status and user operation behavior, including various operation records and data inputs. This data may contain errors, duplicates, or anomalies, requiring sophisticated text understanding and feature extraction for identification and processing. Meanwhile, historical anomaly data cleaning provides records of anomalies or errors that occurred in past system operations, possessing important contextual and background information. By analyzing and comparing historical data, models and algorithms can be built to identify and handle similar anomalies, thereby improving the accuracy of data processing and the stability of the system. This helps prevent the further spread of data confusion and errors, improves database operating efficiency and data analysis accuracy, and provides more reliable and continuous support for the system.

[0026] In the aforementioned abnormal data cleaning method 100 for databases, step S120 involves extracting terminal information data segment-level text understanding feature vectors and historical abnormal data cleaning text global feature vectors from the information data of the front-end processing terminal and the historical abnormal data cleaning data. It should be understood that this provides a solid foundation and support for the stable operation of the database system and data analysis, making data management and anomaly handling more intelligent and reliable, meeting the core requirements of data quality management in the modern big data environment.

[0027] In a specific embodiment of this application, step S120 includes: performing feature extraction on the information data of the front-end processing terminal to obtain a terminal information data segment-level text understanding feature vector; and performing feature extraction on the historical abnormal data cleaning data to obtain a historical abnormal data cleaning text global feature vector.

[0028] It is understandable that the information data from the front-end processing terminal directly reflects system operations and user interactions, including various operation records, input content, and application status. Its diversity and immediacy necessitate efficient data extraction methods. Feature extraction transforms the raw front-end processing terminal information data into multiple terminal information data text understanding feature vectors. These multiple terminal information data text understanding feature vectors not only reflect the specific text content of each data segment but also allow for in-depth analysis and processing using specific text understanding techniques and models. For example, through techniques such as word segmentation, word vectorization, or semantic analysis, text data can be converted into computer-processable structured feature vectors to represent the semantic and content features of each data segment.

[0029] Furthermore, feature extraction from historical anomaly data can transform complex, cleaned historical anomaly data into representative and actionable global feature vectors. By extracting key information and patterns from the data, such as the frequency, duration, and scope of impact of anomalies, the system can better understand and describe the overall characteristics of historical anomaly events. For example, statistical methods, machine learning algorithms, or models specific to a particular domain can be used to extract common anomaly types, key features, and potential influencing factors from large amounts of anomaly data.

[0030] Figure 2 This is a flowchart illustrating the process of extracting features from the information data of the front-end processing terminal to obtain a terminal information data segment-level text understanding feature vector in an abnormal data cleaning method for a database according to an embodiment of this application. Figure 2 As shown, in a specific embodiment of this application, feature extraction is performed on the information data of the front-end processing terminal to obtain terminal information data segment granular text understanding feature vectors, including: S210, data preprocessing is performed on the information data of the front-end processing terminal to obtain multiple terminal information data text understanding feature vectors; S220, feature encoding is performed on the multiple terminal information data text understanding feature vectors to obtain terminal information data segment granular text understanding feature vectors.

[0031] It is understandable that the information data generated by the front-end processing terminal is typically real-time and high-frequency, containing various information such as user interactions and system operations. The diversity and complexity of this data necessitate preprocessing to extract key text understanding feature vectors before it enters the downstream processing flow. Data preprocessing includes, but is not limited to, data cleaning, standardization, normalization, word segmentation, and stop word removal. Through data preprocessing, multiple terminal information data text understanding feature vectors can be extracted from the raw front-end processing terminal information data. These multiple terminal information data text understanding feature vectors reflect the key characteristics of the data, such as operation frequency, user preferences, and common problem patterns, which helps the system to conduct in-depth analysis and understanding of the data.

[0032] Furthermore, feature encoding is performed on the text understanding feature vectors of multiple terminal information data. This aims to convert the textual data features into numerical or vector forms that computers can process and analyze, supporting further data analysis, model training, and decision-making applications. The terminal information data text understanding feature vectors reflect the key semantic and content features of each data segment, such as operation records, input frequency, and user preferences. Through feature encoding, the system can transform multiple terminal information data text understanding feature vectors into feature representations with numerical meaning and computational capabilities, better revealing hidden patterns and relationships within the data.

[0033] Figure 3 This is a flowchart illustrating the process of preprocessing information data from the front-end processing terminal to obtain multiple terminal information data text understanding feature vectors in an abnormal data cleaning method for a database according to an embodiment of this application. For example... Figure 3 As shown, in a specific embodiment of this application, step S210 includes: S211, segmenting the information data of the front-end processing terminal to obtain a front-end processing terminal information word sequence; S212, passing the front-end processing terminal information word sequence through a terminal information data context encoder containing an embedding layer to obtain multiple terminal information data text understanding feature vectors.

[0034] It is understandable that the information data of the front-end processing terminal typically includes user input, system output, and interaction records. This data is diverse and complex, containing rich semantics and information. The main purpose of word segmentation is to divide a continuous text sequence into meaningful words or phrases. Through word segmentation, the original text data can be decomposed into a sequence of words, each word representing a basic unit of data, such as a noun, verb, or adjective. During word segmentation, the complexity and ambiguity of language, as well as the terminology and vocabulary habits of specific domains, must be considered. Therefore, choosing appropriate word segmentation tools or algorithms is crucial, such as rule-based segmentation, statistical segmentation (e.g., Hidden Markov Models, Maximum Entropy Models), or deep learning-based segmentation models (e.g., BiLSTM-CRF Models). These tools and algorithms can effectively segment text data into meaningful word sequences according to different corpora and task requirements.

[0035] Furthermore, when processing terminal information data, word sequences often contain rich contextual information and semantic relationships, such as user intent, operation sequences, and question patterns. This information is crucial for a deep understanding and analysis of user behavior and system operation status. The embedding layer transforms the word sequence into dense real-number vectors by mapping each word to a continuous low-dimensional vector space; this is called word embedding. The selection and training of the embedding layer typically rely on pre-training of the language model or domain-specific datasets to ensure that the generated word embeddings accurately reflect the semantics and content of the terminal information data. The terminal information data context encoder is responsible for encoding and integrating the information of the entire word sequence, thereby generating feature vectors with contextual information. Through the feature vectors generated by the terminal information data context encoder, the system can more comprehensively understand the semantic and content features of each segment of terminal information data, such as word importance, contextual relevance, and the overall sequence structure. Specifically, the embedding layer of the terminal information data context encoder containing the embedding layer is used to map each front-end processing terminal information word in the front-end processing terminal information word sequence into a front-end processing terminal information word embedding vector to obtain a sequence of front-end processing terminal information word embedding vectors; the converter-based BERT model of the terminal information data context encoder containing the embedding layer is used to perform global context semantic encoding on the sequence of front-end processing terminal information word embedding vectors to obtain multiple terminal information data text understanding feature vectors.

[0036] Figure 4 This is a flowchart illustrating the process of feature encoding the text understanding feature vectors of multiple terminal information data to obtain terminal information data segment-level text understanding feature vectors in an abnormal data cleaning method for a database according to an embodiment of this application. Figure 4As shown, in a specific embodiment of this application, step S220 includes: S221, arranging the plurality of terminal information data text understanding feature vectors into a sequence of terminal information segment vectors; S222, passing the sequence of terminal information segment vectors through a terminal information data Bi-LSTM model to obtain terminal information data segment-level text understanding feature vectors.

[0037] It's understandable that terminal information data typically includes records from multiple points in time, such as user interactions and system operations. These records are continuous and ordered in time. By arranging feature vectors into a sequence, the data characteristics and trends at each point in time can be effectively preserved. Serializing feature vectors helps the system capture the dynamic changes and evolutionary trends in terminal information data. For example, user behavior patterns may differ at different times, and the efficiency and response time of system operations will also change over time. Arranging feature vectors in chronological order can effectively reflect these changes, helping the system better understand and analyze the temporal correlations behind the data.

[0038] Furthermore, the sequence of terminal information segment vectors is processed through a Bidirectional Long Short-Term Memory (Bi-LSTM) model for terminal information data, aiming to further enhance the extraction and representation capabilities of granular text understanding features for terminal information data segments. Terminal information data typically contains a large amount of temporal information and complex semantic relationships, such as user interactions and system operations. This information needs to be jointly modeled and predicted at multiple time steps. The Bi-LSTM model is a deep learning model suitable for sequence data processing. By simultaneously considering past and future information at each time point in the sequence, it effectively captures the long-term dependencies of sequence data. The forward LSTM captures information from the beginning of the sequence to the current time point, while the backward LSTM captures information from the end of the sequence to the current time point. The combination of the two can comprehensively understand and encode the semantic and content features of the entire sequence. The granular text understanding feature vectors for terminal information data segments generated by the Bi-LSTM model can more accurately reflect the dynamic changes and evolution trends of terminal information data over time. The model can effectively learn and extract important features at each time point, such as changes in user intent, patterns in operation sequences, and fluctuations in system state. This information is crucial for subsequent data analysis, pattern recognition, and prediction tasks.

[0039] In a specific embodiment of this application, feature extraction of the historical abnormal data cleaning data to obtain a global feature vector of historical abnormal data cleaning text includes: passing the historical abnormal data cleaning data through a converter-based historical abnormal data cleaning text context semantic encoder to obtain multiple historical abnormal data cleaning text semantic feature vectors; and concatenating the multiple historical abnormal data cleaning text semantic feature vectors to obtain a global feature vector of historical abnormal data cleaning text.

[0040] It is understandable that historical anomaly data cleansing text typically contains records of device or system anomaly events, which may have complex relationships and dependencies in both time and context. Transformer-based model structures are widely used in text understanding and sequence modeling tasks due to their ability to process input sequences in parallel. Transformer models can effectively capture long-distance dependencies and exhibit good performance when processing long sequences. This characteristic allows transformer-based semantic encoders to encode the complex semantic information of historical anomaly data cleansing text into high-dimensional semantic feature vectors without information loss. Specifically, the historical anomaly data cleansing text context semantic encoder comprehensively considers the context and semantic information in each anomaly event record, such as the chronological order of events, causal relationships between events, and the importance of events. Through this comprehensive analysis, the encoder can generate feature vectors with rich semantics, which not only reflect the important features of each event record but also capture patterns and anomalous behaviors in the overall dataset. Specifically, the historical abnormal data cleaning data is segmented to obtain a sequence of historical abnormal data cleaning words; the embedding layer of the converter-based historical abnormal data cleaning text context semantic encoder is used to map each historical abnormal data cleaning word in the sequence of historical abnormal data cleaning words into a sequence of historical abnormal data cleaning word embedding vectors to obtain a sequence of historical abnormal data cleaning word embedding vectors; the converter-based BERT model of the converter-based historical abnormal data cleaning text context semantic encoder is used to perform global context semantic encoding on the sequence of historical abnormal data cleaning word embedding vectors to obtain multiple historical abnormal data cleaning text semantic feature vectors.

[0041] Furthermore, multiple semantic feature vectors from historical anomaly data cleaning texts are concatenated to comprehensively integrate the rich information and semantic content contained in each individual feature vector, thereby obtaining a global feature vector for the historical anomaly data cleaning text. Concatenating multiple semantic feature vectors from historical anomaly data cleaning texts can increase the dimensionality and diversity of the data, thus better reflecting the complexity and diversity of the overall dataset. Each semantic feature vector from historical anomaly data cleaning texts may capture specific semantic information from different time points or different event types. By concatenating them, this information can be organically combined to form a more representative and global feature vector representation. Moreover, events in historical anomaly data cleaning texts are usually not isolated; they may have certain time-series relationships or causal connections. For example, the influence of a previous event may lead to the occurrence of subsequent events. By concatenating feature vectors, this relationship can be effectively maintained and expressed, thus providing a more complete and coherent global data perspective.

[0042] In the aforementioned abnormal data cleaning method 100 for databases, step S130 involves obtaining an abnormal data cleaning processing label based on the granular text understanding feature vector of the terminal information data segment and the global feature vector of the historical abnormal data cleaning text. It should be understood that the granular text understanding feature vector of the terminal information data segment reflects detailed behavioral and status information during device operation. Generated through deep learning models such as Bi-LSTM, this feature vector captures the temporal dynamics and user behavior patterns in the operation sequence. For example, it may include key information such as the device's real-time operating status, the frequency of user operations, and system response time, which helps identify normal operations and potential abnormal behaviors. The global feature vector of the historical abnormal data cleaning text integrates previously recorded abnormal events and their handling, providing a comprehensive understanding of the system's historical operating status. This vector not only includes the type, frequency, and scope of impact of abnormal events but also reveals the correlations and patterns between them. Through in-depth analysis of historical data, the system can learn the typical characteristics and possible root causes of abnormal events, providing a reference for future anomaly prediction and handling. By comprehensively utilizing granular text understanding feature vectors from terminal information data segments and global feature vectors from historical anomaly data cleansing text, it is possible to generate and apply anomaly data cleaning and processing labels. For example, the model can determine whether the current operation is abnormal based on the real-time characteristics of terminal information data, and simultaneously infer the severity and urgency of the anomaly event by combining global features from historical data, thereby generating corresponding anomaly processing labels. The generation of anomaly data cleaning and processing labels not only helps in timely response to and handling of current anomalies, but also supports the intelligent optimization and preventative measures formulation of the system during long-term operation. Through continuous monitoring and model updates, the system can continuously optimize the accuracy and efficiency of anomaly detection, improve the stability and security of equipment and systems, and thus effectively reduce operational risks and losses caused by anomaly events.

[0043] In a specific embodiment of this application, step S130 includes: calculating the transition matrix between the terminal information data segment granular text understanding feature vector and the historical abnormal data cleaning text global feature vector to obtain a terminal-historical data association feature matrix; passing the terminal-historical data association feature matrix through a terminal-historical data association convolutional neural network as a feature extractor to obtain a terminal-historical data joint feature vector; performing bidirectional linear manifold modulation based on class compatibility shift on the terminal-historical data joint feature vector to obtain an optimized terminal-historical data joint feature vector; and passing the optimized terminal-historical data joint feature vector through a classifier to obtain a classification result, wherein the classification result is used to represent the abnormal data cleaning processing label.

[0044] It can be understood that the granular text understanding feature vectors of the terminal information data segment capture detailed semantic and state information during device operation. These feature vectors are typically generated using deep learning models, effectively representing the temporal dynamics and user behavior patterns within the operation sequence. Simultaneously, the global text feature vectors of historical anomaly data cleansing integrate the recording and handling of past anomaly events, reflecting the long-term evolution and changing trends of the system's operating state. By calculating the transition matrix, the mutual influence and correlation between the terminal information data feature vectors and the historical anomaly data feature vectors can be quantified. This includes exploring how terminal operation data may lead to or be affected by historical anomaly events, and how anomaly events reflect and influence patterns and behaviors in terminal operations. The calculation of the transition matrix, based on statistical data analysis and machine learning techniques, can identify potential correlations between anomaly events and normal operations, thereby providing the system with more accurate anomaly detection and prediction capabilities.

[0045] Furthermore, the terminal-historical data association feature matrix is ​​processed by a terminal-historical data association convolutional neural network (CNN) acting as a feature extractor. This aims to effectively capture and express the complex correlation features between terminal device operation data and historical anomaly event data using deep learning techniques, thereby generating a joint feature vector of terminal-historical data. The terminal-historical data association feature matrix contains the correlation information between terminal device operation data and historical anomaly event data, calculated using statistical and machine learning techniques. This matrix reflects the correlation and influence between data across different time periods and operational states, helping the system understand the potential causal relationship between device operation behavior and the occurrence of anomalies. The terminal-historical data association CNN, acting as a feature extractor, possesses the ability to handle complex data correlations. Here, the CNN can effectively extract spatial and temporal series features, making it suitable for handling dynamic changes in terminal operation data and long-term patterns in historical anomaly event data. By training the CNN model, the information in the terminal-historical data association feature matrix can be transformed into a higher-level, more abstract joint feature representation. This multi-level feature extraction process can not only capture local patterns between equipment operation and abnormal events, but also identify and analyze global data trends and abnormal behaviors, providing the system with a more comprehensive and accurate data analysis foundation.

[0046] Furthermore, in the technical solution of this application, it is considered that the information data and historical abnormal data of the front-end processing terminal may come from different data sources and have different data distribution characteristics, leading to local differences in feature vectors. During word segmentation and context encoding, due to the limitations of language models and word segmentation algorithms, they may not be able to fully and accurately reflect the semantic information of the original text, resulting in local biases in feature vectors. Although the Bi-LSTM model can capture the contextual information of the text, it may not be able to fully capture all long-distance dependencies, especially in long texts or texts with complex structures. The historical abnormal data cleaning text context semantic encoder based on the converter may exhibit different encoding effects on different data, leading to local distribution differences in feature vectors. When concatenating multiple feature vectors into a global feature vector, the contributions of different feature vectors may not be reasonably weighted, leading to imbalances in local features. This results in local distribution differences in the joint feature vector of the terminal-historical data, which in turn causes a class compatibility shift relative to the predetermined class label when classified by the classifier, affecting the classification iteration effect. Based on this, the joint feature vector of the terminal-historical data is subjected to bidirectional linear manifold modulation based on class compatibility shift.

[0047] The process of applying bidirectional linear manifold modulation based on class compatibility shift to the joint feature vector of terminal-historical data to obtain an optimized joint feature vector of terminal-historical data includes: calculating the joint autocorrelation matrix of terminal-historical data between the joint feature vector of terminal-historical data and its transpose; performing matrix multiplication of the weight parameter matrix of the classifier with the joint autocorrelation matrix of terminal-historical data to obtain a weight parameter-modulated classification autocorrelation matrix; calculating the correlation perception factor between any two row vectors in the weight parameter-modulated classification autocorrelation matrix to obtain a metric correlation perception matrix; and multiplying the metric correlation perception matrix with the joint feature vector of terminal-historical data to obtain the optimized joint feature vector of terminal-historical data.

[0048] Here, the terminal-historical data joint feature vector is subjected to bidirectional linear manifold modulation based on class compatibility shift to obtain an optimized terminal-historical data joint feature vector, including: processing the terminal-historical data joint feature vector with the following optimization formula to obtain an optimized terminal-historical data joint feature vector;

[0049] The optimization formula is as follows:

[0050]

[0051] Among them, V cV is the joint feature vector of the terminal-historical data. c T M is the transpose of the terminal-historical data joint feature vector. c It is a terminal-historical data joint autocorrelation matrix, M p M is the weight parameter matrix of the classifier. t It is the classification autocorrelation matrix modulated by weight parameters, V i V is the row vector of the i-th row in the weight parameter modulation classification autocorrelation matrix. j V is the row vector of the j-th row in the weight parameter modulation classification autocorrelation matrix. j T It is the transpose of the row vector of the j-th row in the weight parameter modulation classification autocorrelation matrix, where L is the total number of rows in the weight parameter modulation classification autocorrelation matrix, and m is the total number of rows in the matrix. i,j It is a row vector V i and row vector V j The correlation coefficient between them, ||·||2 is the L2 norm of the feature vectors, Let M′ represent matrix multiplication, M′ be the metric correlation perception matrix, R be the set of real numbers, and V′ be the optimized terminal-historical data joint feature vector.

[0052] In other words, in the technical solution of this application, considering the local distribution differences in the joint feature vector of terminal-historical data, which cause a class compatibility shift relative to the predetermined class label when the classifier performs classification judgment, thus affecting the classification iteration effect, the technical solution of this application expands the association based on the spatial sub-dimensional complexity of the high-dimensional feature space of the feature distribution, based on the self-correlation dimension of the joint feature vector of terminal-historical data, to obtain the joint autocorrelation matrix of terminal-historical data. Furthermore, the joint autocorrelation matrix of terminal-historical data is subjected to a relevance perception measurement based on weight learning parameters. By introducing weight learning parameters, the local distribution differences of different joint feature vectors of terminal-historical data can be effectively quantified and adjusted, thereby reducing the class compatibility shift caused by these local distribution differences when the classifier performs classification judgment.

[0053] In particular, the optimized terminal-historical data joint feature vector integrates the correlation features of terminal device operation data and historical abnormal event data, possessing high information richness and expressive power. After being processed by a deep learning model (such as a convolutional neural network or a recurrent neural network), the optimized terminal-historical data joint feature vector can effectively capture the real-time status of device operation and the potential patterns of abnormal behavior, forming a comprehensive and accurate understanding of the system's operating status. The classification results serve as labels for abnormal data cleaning and processing, providing guidance and operability. It can not only be used for real-time monitoring and identification of abnormal behavior in device operation, but also support decision-makers in making timely and effective responses when handling abnormal events. For example, the system can automatically trigger preset abnormal handling procedures based on the classification results, or provide operators with necessary information and suggestions so that they can quickly take appropriate measures to fix problems or adjust operating strategies.

[0054] In summary, the embodiments of this application first obtain information data from the front-end processing terminal and historical abnormal data cleaning data, then use deep learning technology to extract features and perform correlation analysis on the two, and finally use a classifier to obtain abnormal data cleaning processing labels, so as to effectively improve the efficiency and quality of data cleaning, thereby improving the detection speed of abnormal data and reducing the need for manual intervention.

[0055] Figure 5 This is a block diagram of an abnormal data cleaning apparatus for a database according to an embodiment of this application. Figure 5 As shown, the abnormal data cleaning apparatus 100 for a database according to an embodiment of this application includes: a database data information acquisition module 110, used to acquire information data of a front-end processing terminal and historical abnormal data cleaning data; a database data information extraction module 120, used to extract terminal information data segment granular text understanding feature vectors and historical abnormal data cleaning text global feature vectors from the information data of the front-end processing terminal and the historical abnormal data cleaning data; and an abnormal data cleaning processing label representation module 130, used to obtain abnormal data cleaning processing labels based on the terminal information data segment granular text understanding feature vectors and the historical abnormal data cleaning text global feature vectors.

[0056] As described above, the abnormal data cleaning device 100 for databases according to the embodiments of this application can be implemented in various terminal devices. In one example, the abnormal data cleaning device 100 for databases can be integrated into the terminal device as a software module and / or a hardware module. For example, the abnormal data cleaning device 100 for databases can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the abnormal data cleaning device 100 for databases can also be one of many hardware modules of the terminal device.

[0057] Alternatively, in another example, the abnormal data cleaning device 100 for the database and the terminal device can also be separate devices, and the abnormal data cleaning device 100 for the database can be connected to the terminal device via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0058] Here, those skilled in the art will understand that the specific operations of each step in the above-described abnormal data cleaning apparatus for databases have been referenced above. Figures 1 to 4 The description of the abnormal data cleaning methods for databases is detailed here, and therefore, its repeated description will be omitted.

[0059] Below, for reference Figure 6 This describes an electronic device according to embodiments of the present application.

[0060] Figure 6 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0061] like Figure 6 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are interconnected via the bus 17. The input device 11 and the output device 16 are connected to the bus 17 via the input interface 14 and the output interface 15, respectively, and are thus connected to other components of the electronic device 10.

[0062] Specifically, input device 11 receives input information from the outside and transmits the input information to central processing unit 13 through input interface 12; central processing unit 13 processes the input information based on computer-executable instructions stored in memory 14 to generate output information, temporarily or permanently stores the output information in memory 14, and then transmits the output information to output device 16 through output interface 15; output device 16 outputs the output information to the outside of electronic device 10 for user use.

[0063] In one embodiment, Figure 6 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to execute any of the abnormal data cleaning methods for databases described in the above embodiments.

[0064] According to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0065] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0066] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of this application, and this application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this application, and these modifications and improvements are also considered to be within the scope of protection of this application.

Claims

1. A method for cleaning abnormal data in a database, characterized in that, include: Acquire information data from the front-end processing terminal and clean historical abnormal data; Extract terminal information data segment granular text understanding feature vectors and historical abnormal data cleaning text global feature vectors from the information data of the front-end processing terminal and the historical abnormal data cleaning data. Based on the granular text understanding feature vector of the terminal information data segment and the global feature vector of the historical abnormal data cleaning text, an abnormal data cleaning processing label is obtained.

2. The abnormal data cleaning method for databases according to claim 1, characterized in that, Extract terminal information data segment-level text understanding feature vectors and historical anomaly data cleaning text global feature vectors from the information data of the front-end processing terminal and the historical anomaly data cleaning data, including: Feature extraction is performed on the information data of the front-end processing terminal to obtain a terminal information data segment granular text understanding feature vector; Feature extraction is performed on the historical abnormal data cleaning data to obtain a global feature vector of the historical abnormal data cleaning text.

3. The abnormal data cleaning method for databases according to claim 2, characterized in that, Feature extraction is performed on the information data of the front-end processing terminal to obtain a terminal information data segment-level text understanding feature vector, including: The information data of the front-end processing terminal is preprocessed to obtain multiple terminal information data text understanding feature vectors; Feature encoding is performed on the text understanding feature vectors of the multiple terminal information data to obtain the granular text understanding feature vectors of the terminal information data segments.

4. The abnormal data cleaning method for databases according to claim 3, characterized in that, The information data from the front-end processing terminal is preprocessed to obtain multiple terminal information data text understanding feature vectors, including: The information data of the front-end processing terminal is segmented into words to obtain the information word sequence of the front-end processing terminal; The sequence of terminal information words processed by the foreground is passed through a terminal information data context encoder containing an embedding layer to obtain multiple terminal information data text understanding feature vectors.

5. The abnormal data cleaning method for databases according to claim 4, characterized in that, Feature encoding is performed on the multiple terminal information data text understanding feature vectors to obtain terminal information data segment-level text understanding feature vectors, including: Arrange the multiple terminal information data text understanding feature vectors into a sequence of terminal information segment vectors; The sequence of terminal information segment vectors is processed through a terminal information data Bi-LSTM model to obtain terminal information data segment granular text understanding feature vectors.

6. The abnormal data cleaning method for databases according to claim 5, characterized in that, Feature extraction is performed on the historical anomaly data cleaning data to obtain a global feature vector of the historical anomaly data cleaning text, including: The historical anomaly data cleaning data is passed through a converter-based historical anomaly data cleaning text context semantic encoder to obtain multiple historical anomaly data cleaning text semantic feature vectors. The semantic feature vectors of the multiple historical abnormal data cleaning texts are concatenated to obtain the global feature vector of the historical abnormal data cleaning text.

7. The abnormal data cleaning method for databases according to claim 6, characterized in that, Based on the terminal information data segment granular text understanding feature vector and the historical abnormal data cleaning text global feature vector, abnormal data cleaning processing labels are obtained, including: Calculate the transition matrix between the granular text understanding feature vector of the terminal information data segment and the global feature vector of the historical abnormal data cleaning text to obtain the terminal-historical data association feature matrix; The terminal-historical data association feature matrix is ​​passed through a terminal-historical data association convolutional neural network, which acts as a feature extractor, to obtain a joint feature vector of terminal-historical data. The terminal-historical data joint feature vector is subjected to bidirectional linear manifold modulation based on class compatibility shift to obtain an optimized terminal-historical data joint feature vector; The optimized terminal-historical data joint feature vector is passed through a classifier to obtain a classification result, which is used to represent the label for abnormal data cleaning and processing.

8. The abnormal data cleaning method for databases according to claim 7, characterized in that, The terminal-historical data joint feature vector is subjected to bidirectional linear manifold modulation based on class compatibility shift to obtain an optimized terminal-historical data joint feature vector, including: Calculate the terminal-historical data joint autocorrelation matrix between the terminal-historical data joint feature vector and its transpose vector; The weight parameter matrix of the classifier is multiplied with the terminal-historical data joint autocorrelation matrix to obtain the weight parameter modulated classification autocorrelation matrix; Calculate the correlation-perceived factor between any two row vectors in the weight parameter modulation classification autocorrelation matrix to obtain the correlation-perceived matrix. The correlation perception matrix is ​​multiplied by the terminal-historical data joint feature vector to obtain the optimized terminal-historical data joint feature vector.

9. An anomalous data cleaning device for a database, characterized in that, include: The database data information acquisition module is used to acquire information data from the front-end processing terminal and historical abnormal data cleaning data; The database data information extraction module is used to extract terminal information data segment granular text understanding feature vectors and historical abnormal data cleaning text global feature vectors from the information data of the front-end processing terminal and the information data of the historical abnormal data cleaning. The abnormal data cleaning and processing label representation module is used to obtain abnormal data cleaning and processing labels based on the granular text understanding feature vector of the terminal information data segment and the global feature vector of the historical abnormal data cleaning text.

10. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the abnormal data cleaning method for a database as described in any one of claims 1 to 8.

Images