Dimensionality Reduction Method, Anomaly Detection Method, Device, System and Equipment for Annotated Data

Abstract

The present application relates to the technical field of data processing, and in particular to a method for dimension reduction of labeled data, a quality detection method, an apparatus, a system, and a device. The method includes: inputting a labeled data set to be evaluated into a convolutional neural network, performing feature extraction and downsampling processing respectively to obtain a corresponding channel map and an original feature map; using an incremental principal component analysis method introducing a regularization term to perform dimension reduction processing on the pixel values in the channel map to obtain a feature map regarding a covariance matrix; and fusing the feature map regarding the covariance matrix with the original feature map to obtain the dimension-reduced labeled data set. By using this method, the dimension reduction efficiency of high-dimensional labeled data and the anomaly detection efficiency can be improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a method for dimensionality reduction of labeled data, an anomaly detection method, a device, a system, and a device. Background Art

[0002] In the field of intelligent driving, the quality of data annotation plays a crucial role in the training of artificial intelligence models. The higher the quality of data annotation, the higher the quality of the trained artificial intelligence model.

[0003] Common methods for detecting anomalies in labeled data are difficult to apply to high-dimensional labeled data, which is likely to cause the curse of dimensionality, leading to an increase in computational complexity and a decrease in performance.

[0004] Currently, the existing dimensionality reduction methods for labeled data usually adopt the traditional principal component analysis method. However, when this method is applied to large-scale data sets, there are problems of high computational complexity and easy loss of information, resulting in a decrease in the efficiency of data dimensionality reduction and further a decrease in the efficiency of anomaly detection of labeled data. Summary of the Invention

[0005] Based on this, a method for dimensionality reduction of labeled data, an anomaly detection method, a device, a system, and a device are provided to improve the dimensionality reduction efficiency and anomaly detection efficiency of high-dimensional labeled data.

[0006] In a first aspect, a method for dimensionality reduction of labeled data is provided, and the method includes:

[0007] Input the labeled data set to be evaluated into a convolutional neural network, and perform feature extraction and downsampling processing respectively to obtain corresponding channel maps and original feature maps;

[0008] Adopt an incremental principal component analysis method with a regularization term to perform dimensionality reduction processing on the pixel values in the channel map to obtain a feature map of the covariance matrix;

[0009] Fuse the feature map of the covariance matrix with the original feature map to obtain the dimensionality-reduced labeled data set.

[0010] Combined with the first aspect, in the first first implementable manner of the first aspect, the step of adopting an incremental principal component analysis method with a regularization term to perform dimensionality reduction processing on the pixel values in the channel map to obtain a feature map of the covariance matrix includes:

[0011] Divide the pixel values in the channel map into several batches of pixel values, and obtain an optimized objective function after introducing a regularization term;

[0012] According to the optimized objective function, calculate the principal components and eigenvalues of each batch of pixel values one by one;

[0013] Accumulate the principal components and eigenvalues calculated from each of the batch pixel values to obtain corresponding final principal components and final eigenvalues;

[0014] Project the pixel values in the channel map onto the final principal components to obtain the eigenmap with respect to the covariance matrix.

[0015] Combined with the first implementable manner of the first aspect, in the second implementable manner of the first aspect, the steps of obtaining the optimized objective function after introducing the regularization term include:

[0016] Obtain an initial objective function, where the initial objective function represents obtaining a projection matrix that maximizes the variance of the pixel values in the channel map after projection;

[0017] Obtain a preset regularization parameter and a regularization term, where the regularization term includes the sum of the squares of all elements in the projection matrix;

[0018] Adjust the initial objective function according to the regularization parameter and the regularization term to obtain the optimized objective function.

[0019] Combined with the first implementable manner of the first aspect, in the third implementable manner of the first aspect, the steps of calculating the principal components and eigenvalues of each of the batch pixel values one by one according to the optimized objective function include:

[0020] Based on the first batch of pixel values among the several batches of pixel values, solve the optimized objective function to obtain the pixel variance of the first batch of pixel values;

[0021] Calculate the pixel mean of the first batch of pixel values, and calculate the covariance matrix of the first batch of pixel values according to the pixel variance and the pixel mean;

[0022] Perform eigenvalue decomposition on the covariance matrix to obtain the principal components and eigenvalues corresponding to the first batch of pixel values;

[0023] Update the principal components and eigenvalues of the remaining batch pixel values one by one through the principal components and eigenvalues corresponding to the first batch of pixel values.

[0024] In a second aspect, there is provided an abnormal detection method for labeled data, the method including:

[0025] Obtain a labeled data set to be evaluated;

[0026] By means of the steps of the dimensionality reduction method for labeled data described in the first aspect or in any one of the first to third implementable ways in combination with the first aspect, perform dimensionality reduction processing on the to-be-evaluated labeled data set to obtain a dimensionality-reduced labeled data set;

[0027] Input the dimensionality-reduced labeled data set into the trained isolation forest anomaly detection model for processing to obtain corresponding anomaly detection results.

[0028] In combination with the second aspect, in the first implementable way of the second aspect, before the step of inputting the dimensionality-reduced labeled data set into the trained isolation forest anomaly detection model for processing, the method further includes:

[0029] Obtain the original training data set with dimensionality reduction and labeled information, and process the original training data set by using a multi-scale expression method to obtain a multi-scale training data set;

[0030] Iteratively train the isolation forest anomaly detection model by using the multi-scale training data set to obtain the trained isolation forest anomaly detection model.

[0031] In combination with the first implementable way of the second aspect, in the second implementable way of the second aspect, the step of processing the original training data set by using a multi-scale expression method to obtain a multi-scale training data set includes:

[0032] Take the image pyramid of the original training data set as the multi-scale training data set; or,

[0033] Use a number of windows with different scales to slide on the original training data set and capture local information at different scales to obtain the multi-scale training data set; or,

[0034] Extract features from the original training data set by using a multi-scale feature extractor to obtain the multi-scale training data set; or,

[0035] Perform several different-scale decompositions on the original training data set to obtain several sub-data sets, and take the set of the several sub-data sets as the multi-scale training data set; or,

[0036] Take the versions of the original training data set at several different resolutions as the multi-scale training data set; or,

[0037] Perform several different-scale decompositions on the original training data set to obtain several sub-data sets, perform arbitrary combinations on the sub-data sets at different scales to obtain combined data sets, and take the set of all the combined data sets obtained by combination as the multi-scale training data set.

[0038] In a third aspect, a dimensionality reduction device for labeled data is provided. The device includes:

[0039] A feature extraction unit, configured to input a labeled data set to be evaluated into a convolutional neural network, perform feature extraction and downsampling processing respectively, and obtain corresponding channel maps and original feature maps;

[0040] A dimensionality reduction unit, configured to perform dimensionality reduction processing on the pixel values in the channel maps by using an incremental principal component analysis method introducing a regularization term, and obtain a feature map regarding the covariance matrix;

[0041] A fusion unit, configured to fuse the feature map regarding the covariance matrix with the original feature map to obtain the dimensionality-reduced labeled data set.

[0042] In a fourth aspect, an anomaly detection system for labeled data is provided. The system includes:

[0043] A data set acquisition module, configured to acquire a labeled data set to be evaluated;

[0044] The dimensionality reduction device for labeled data as described in the third aspect, configured to perform dimensionality reduction processing on the labeled data set to be evaluated to obtain a dimensionality-reduced labeled data set;

[0045] A data set anomaly detection module, configured to input the dimensionality-reduced labeled data set into a trained isolation forest anomaly detection model for processing to obtain corresponding anomaly detection results.

[0046] In a fifth aspect, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the dimensionality reduction method for labeled data as described in the first aspect or any one of the first to third implementable embodiments combined with the first aspect are implemented.

[0047] The above-mentioned dimensionality reduction method, anomaly detection method, device, system, and equipment for labeled data input the labeled data set to be evaluated into a convolutional neural network, perform feature extraction and downsampling respectively to obtain corresponding channel maps and original feature maps; adopt an incremental principal component analysis method introducing a regularization term to perform dimensionality reduction on the pixel values in the channel maps to obtain a feature map regarding the covariance matrix; fuse the feature map regarding the covariance matrix with the original feature map to obtain a dimensionality-reduced labeled data set. Compared with the prior art, the beneficial effects of the dimensionality reduction method for labeled data in this application are as follows: not only can data dimensionality reduction be achieved to ensure the quality detection efficiency of labeled data, but also compared with existing dimensionality reduction methods, the convolutional neural network is applicable to complex image processing tasks, and the incremental principal component analysis method can be applicable to large-scale data sets to solve the problems of high computational complexity and loss of important information, while the introduction of the regularization term can improve the stability of the processing task, thereby improving the dimensionality reduction efficiency of high-dimensional labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 FIG. is an application environment diagram of a quality detection method for labeled data in an embodiment;

[0049] Figure 2 FIG. is a schematic diagram of the original features of the iris data set exemplified in an embodiment;

[0050] Figure 3 FIG. is a schematic flowchart of a dimensionality reduction method for labeled data in an embodiment;

[0051] Figure 4 FIG. is a schematic diagram of the features after dimensionality reduction of the iris data set exemplified in an embodiment;

[0052] Figure 5 FIG. is a schematic flowchart of a quality detection method for labeled data in an embodiment;

[0053] Figure 6 FIG. is a distribution diagram of anomaly scores in an embodiment;

[0054] Figure 7 FIG. is a schematic diagram of a consistency evaluation result in an embodiment;

[0055] Figure 8 FIG. is a structural block diagram of a dimensionality reduction device for labeled data in an embodiment;

[0056] Figure 9 FIG. is a structural block diagram of a quality detection system for labeled data in an embodiment;

[0057] Figure 10 FIG. is a structural block diagram of a quality detection system for labeled data in another embodiment;

[0058] Figure 11 The internal structure diagram of a computer device in an embodiment. Specific implementation manners

[0059] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that the diagrams provided in this embodiment only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape and size of the components in actual implementation. The types, numbers and ratios of the components in actual implementation may be arbitrarily changed, and the component layout type may also be more complex.

[0061] The structures, ratios, sizes, etc. shown in the drawings of this specification are only used to cooperate with the content disclosed in the specification for those skilled in this technology to understand and read, and are not used to limit the limiting conditions under which the present application can be implemented. Therefore, they do not have technical essential meanings. Any modification of the structure, change of the ratio relationship or adjustment of the size, without affecting the effects that the present application can produce and the objectives that can be achieved, should still fall within the scope that can be covered by the technical content disclosed in the present application.

[0062] The orientation or positional relationship indicated by terms such as "upper", "lower", "left", "right", "middle", "longitudinal", "transverse", "horizontal", "inner", "outer", "radial", "circumferential", etc. cited in this specification is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of simplified description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation to the present application. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0063] The dimensionality reduction method for labeled data provided by the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. By inputting the labeled data set to be evaluated into the convolutional neural network, feature extraction and downsampling processes are respectively performed to obtain the corresponding channel map and the original feature map; the incremental principal component analysis method introducing a regularization term is used to perform dimensionality reduction processing on the pixel values in the channel map to obtain the feature map regarding the covariance matrix; the feature map regarding the covariance matrix is fused with the original feature map to obtain the dimensionality-reduced labeled data set, which can not only achieve dimensionality reduction of data, thereby overcoming the problem of data dimensionality disaster to a certain extent and improving the anomaly detection efficiency for high-dimensional labeled data; but also be applicable to large-scale data sets, solve the problems of high computational complexity and loss of important information, improve the stability of the processing task, thereby improving the dimensionality reduction efficiency of high-dimensional labeled data, and further ensuring the anomaly detection efficiency for high-dimensional labeled data. Among them, the terminal 102 can be but is not limited to various personal computers, laptop computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0064] First, collect the labeled data set that needs to be subjected to anomaly detection. Among them, the labeled data set represents a set of multiple labeled data, and these labeled data represent the data that has been labeled and has label information. Taking the iris data set as an example, the features of the labeled data set to be evaluated are as Figure 2 shown.

[0065] It can be seen Figure 2 that the data features in it show a scattered trend, which means that there may be a lack of obvious correlation or pattern between the features. This will make it difficult for the model to capture meaningful feature relationships during the learning or training process, thereby reducing the performance of the model. For example, in a high-dimensional space, the scattered feature distribution may make it difficult for the model to distinguish different categories or find the potential structure in the data. The noise points in the data may be caused by measurement errors, outliers or other interference factors.

[0066] These noise points may have a negative impact on the robustness and generalization of the model. The model may try to adapt to these noise points, resulting in poor prediction effects on new data. Therefore, it is necessary to perform feature dimensionality reduction on the labeled data set so that the model can focus more on the most important information in the data, thereby alleviating the problem of dimensionality disaster and improving the data processing efficiency. In addition, feature alignment also helps to improve the generalization performance of the model, enabling it to better adapt to new and unprocessed data.

[0067] Currently, existing data dimensionality reduction usually uses the traditional Principal Component Analysis (PCA) to perform dimensionality reduction on the labeled dataset. The goal of PCA is to find the direction with the largest variance and project the data onto this direction. The projected data retains as much of the original data information as possible. PCA converts the original data into a set of new variables through a linear transformation, and these new variables are called principal components.

[0068] PCA generally includes the following steps: (1) Standardize the data: First, standardize the data so that each feature has a similar scale, avoiding some features dominating in the principal component analysis due to their numerical ranges; (2) Calculate the covariance matrix: First, calculate the sample mean: Then calculate the sample variance: Finally, calculate the covariance matrix: The covariance matrix describes the correlation between features in the dataset. Therefore, the relationship between features is determined by calculating the covariance matrix of the dataset; (3) Calculate eigenvalues and eigenvectors: By performing eigenvalue decomposition on the covariance matrix, find its eigenvalues and corresponding eigenvectors. The eigenvalues represent the importance of each eigenvector, and the eigenvectors represent the directions of the principal components; (4) Select the principal components: The dimensions corresponding to the eigenvectors with larger eigenvalues are considered to contain the most important information in the data. Therefore, select the first k eigenvectors (principal components) to construct a new feature space; (5) Project the data: Project the original data onto the selected principal components to obtain the dimensionality-reduced dataset.

[0069] Since the traditional PCA algorithm has the following disadvantages for dimensionality reduction of image data: (1) High computational complexity: For large-scale datasets, the computational complexity of PCA is relatively high, requiring a large amount of computing resources and time; (2) Information loss: The goal of PCA is to maximize the variance of the data, which may lead to the loss of some original data information in the low-dimensional space. Although PCA will select to retain the most important variances, it does not guarantee that the features corresponding to these variances are the most important features; (3) Sensitive to data preprocessing: PCA is very sensitive to data preprocessing. For example, if the scales of the data are different or not centered around zero, then the output of PCA may be affected.

[0070] Therefore, this application proposes a method for dimensionality reduction of labeled data. In one embodiment, as Figure 3 shown, taking the application of this method to Figure 1 the terminal as an example for illustration, it includes the following steps:

[0071] Step 202, input the labeled dataset to be evaluated into the convolutional neural network, and perform feature extraction and downsampling processing respectively to obtain the corresponding channel map and original feature map;

[0072] Step 204: Use incremental principal component analysis with a regularization term to perform dimensionality reduction on the pixel values in the channel map, obtaining a feature map regarding the covariance matrix.

[0073] Step 206: Fuse the feature map regarding the covariance matrix with the original feature map to obtain the dimensionality-reduced labeled data set.

[0074] In a specific implementation manner, step 204 may include: dividing the pixel values in the channel map into several batches of pixel values, and obtaining an optimized objective function with a regularization term; according to the optimized objective function, calculating the principal components and eigenvalues of each batch of pixel values one by one; accumulating the principal components and eigenvalues calculated for each batch of pixel values to obtain corresponding final principal components and final eigenvalues; projecting the pixel values in the channel map onto the final principal components to obtain the feature map regarding the covariance matrix.

[0075] Further, the step of obtaining the optimized objective function with a regularization term includes: obtaining an initial objective function, where the initial objective function represents obtaining a projection matrix that maximizes the variance of the pixel values in the channel map after projection; obtaining a preset regularization parameter and a regularization term, where the regularization term includes the sum of the squares of all elements in the projection matrix; adjusting the initial objective function according to the regularization parameter and the regularization term to obtain the optimized objective function.

[0076] Furthermore, the step of calculating the principal components and eigenvalues of each batch of pixel values one by one according to the optimized objective function includes: solving the optimized objective function based on the first batch of pixel values among the several batches of pixel values to obtain the pixel variance of the first batch of pixel values; calculating the pixel mean of the first batch of pixel values, and calculating the covariance matrix of the first batch of pixel values according to the pixel variance and the pixel mean; performing eigenvalue decomposition on the covariance matrix to obtain the principal components and eigenvalues corresponding to the first batch of pixel values; updating the principal components and eigenvalues of the remaining batches of pixel values one by one through the principal components and eigenvalues corresponding to the first batch of pixel values.

[0077] It should be noted that, compared with the traditional PCA algorithm, the incremental PCA algorithm allows data to be processed step by step rather than the entire dataset at once, so it is particularly suitable for processing large-scale data. The incremental PCA algorithm generally includes the following steps: (1) Data segmentation: The large-scale dataset is divided into several small batches, which can be divided according to time order or by random sampling; (2) Initialization: For the first small batch of data, the covariance matrix is calculated and eigenvalue decomposition is performed to obtain the principal components and corresponding eigenvalues; (3) Step-by-step update: For subsequent small batches, instead of recalculating the entire covariance matrix, the principal components and eigenvalues are gradually updated by using the previous results, which can significantly reduce the computational cost; (4) Cumulative results: For each small batch, the updated principal components and eigenvalues can be accumulated to form the final principal components and eigenvalues; (5) Dimensionality reduction projection: By using the updated principal components, new data can be mapped to a low-dimensional space to achieve dimensionality reduction.

[0078] The traditional PCA algorithm selects the principal components by maximizing the variance of the data, but this may lead to being too sensitive to the noise in the data. Introducing a regularization term in the incremental PCA algorithm can constrain the principal components during the optimization process, thereby improving the robustness to noise and outliers and reducing instability.

[0079] Among them, the steps of introducing the regularization term can include: (1) Determining the objective function: The goal of the PCA algorithm is to find the projection direction to maximize the variance of the projected data. The traditional PCA optimization goal is to find a projection matrix W to maximize the variance of the data X after projection, that is: {maximize}\{Var}(XW), where {Var}(XW) represents the variance and {maximize} represents the maximization operation; after introducing the regularization term, the objective function becomes: {maximize}\{Var}(XW)-λ*{RegularizationTerm}(W), where λ is the regularization parameter and {RegularizationTerm}(W) is the regularization term; (2) Selecting the regularization term: The regularization term can be selected in different forms, specifically depending on the understanding of the problem and the assumptions about the data, and also depending on which characteristics are desired to be constrained during the optimization process. Common regularization terms include L1 regularization and L2 regularization. For principal component analysis, L2 regularization is usually used, and its form is the sum of the squares of the principal components, that is: {RegularizationTerm}(W)=|W| 2 where, |W| 2Denotes the square of the Frobenius norm of the projection matrix W, that is, the sum of the squares of all elements in the projection matrix; (3) Optimization problem: After introducing the regularization term, the optimization problem becomes: {maximize} {Var}(XW) - λ|W| 2 , which is an optimization problem with constraints and can be solved using methods such as the Lagrange multiplier method; (4) Adjust the regularization parameter: The choice of the regularization parameter λ is crucial as it controls the influence degree of the regularization term. Through methods such as cross-validation, a λ can be selected that enables the model to perform well on both the training set and the validation set.

[0080] For image data, especially when dealing with complex visual features, convolutional neural networks can usually provide better performance. CNNs can learn the local structures and abstract features in images and are suitable for more complex image processing tasks. Combining CNNs and the incremental PCA algorithm can, to a certain extent, address the problem of important information loss in PCA; this combination typically involves nesting the CNN and the incremental PCA algorithm together to fully utilize their respective advantages, namely: first, use the convolutional and pooling layers of the CNN to extract features from the labeled dataset, then perform dimensionality reduction on the extracted features using incremental PCA, and during the dimensionality reduction process, introduce a regularization term to constrain the principal components. Finally, fuse the feature maps of the covariance matrix obtained after dimensionality reduction with the feature maps corresponding to the labeled dataset that has not undergone dimensionality reduction to obtain the labeled dataset after dimensionality reduction.

[0081] Specifically, it includes the following steps: (1) Feature extraction: (1.1) CNN: Use the convolutional and pooling layers of the CNN to extract features from the image. Through multiple convolutional layers, the network can learn the local and global features of the image and form high-level representations; (1.2) Dimensionality reduction: Apply the incremental PCA algorithm with a regularization term to the features extracted by the CNN for dimensionality reduction. The purpose of this step is to reduce the dimensionality of the features while retaining the most important features; since the features extracted by the CNN may be high-dimensional, dimensionality reduction can help reduce the computational burden and the complexity of the model; (2) Combining PCA and convolutional layers: (2.1) Calculate the local covariance matrix: On the feature map of the CNN, calculate the covariance matrix of the local region, which can be achieved by moving a sliding window on the feature map and calculating the covariance matrix within each window; (2.2) Feature fusion: Fuse the obtained local covariance matrix with the original feature map, which can be achieved by overlaying the information of the covariance matrix with the original feature map or by weighted summation. This operation helps introduce the linear transformation of PCA into the CNN to better capture the correlation of the data.

[0082] The advantage of the method combining CNN and PCA is that CNN can extract high-level features of images, while PCA can perform more effective dimensionality reduction on these features. This combination helps to comprehensively utilize the advantages of the two methods, improve the performance and robustness of the model, and at the same time reduce the information loss that may occur in traditional PCA. Among them, the specific effect may depend on the nature of the dataset and the requirements of the task.

[0083] Taking Figure 2 the aforementioned iris data as an example, the characteristics of the labeled data after dimensionality reduction are finally obtained as Figure 4 shown. It can be seen from Figure 4 that the improved PCA algorithm projects the original data onto the most important principal components by finding the most important principal components in the data, which can eliminate the redundant information between features and retain the main data variations; reducing redundancy helps the model better understand the structure of the data, avoid overfitting, and increase the attention to key features. By selecting the most important principal components and mapping the high-dimensional data to a low-dimensional space, this helps to solve the curse of dimensionality problem and reduce the computational burden during model training and inference; at the same time, reducing the dimension also makes it easier to visualize the data, which helps to intuitively understand the data distribution and structure.

[0084] In summary, the dimensionality reduction method for labeled data in this application combines the incremental PCA algorithm with a convolutional neural network by introducing a regularization term to perform dimensionality reduction on the labeled data. It can not only achieve dimensionality reduction of the data to ensure the quality detection efficiency of the labeled data, but also compared with the existing dimensionality reduction methods, the convolutional neural network is suitable for complex image processing tasks, and the incremental principal component analysis method can be applied to large-scale datasets to solve the problems of high computational complexity and loss of important information. The introduction of the regularization term can improve the stability of the processing task, thereby improving the dimensionality reduction efficiency of high-dimensional labeled data.

[0085] In one embodiment, as Figure 5 shown, an anomaly detection method for labeled data is provided, including the following steps:

[0086] Step 302, obtain the labeled dataset to be evaluated;

[0087] Step 304, perform dimensionality reduction processing on the labeled dataset to be evaluated through the steps of the dimensionality reduction method for labeled data described in the foregoing embodiment to obtain a dimensionality-reduced labeled dataset;

[0088] Step 306, input the dimensionality-reduced labeled dataset into a trained isolation forest anomaly detection model for processing to obtain corresponding anomaly detection results.

[0089] It should be noted that the specific limitations regarding step 304 can be referred to the specific description of the dimensionality reduction method for labeled data in the above text, and will not be elaborated here.

[0090] Before performing step 206, it is necessary to select and train a model, which may include the following steps: (1) Data preparation: a dataset with labeled information after dimensionality reduction processing, ensuring that the dataset contains normal and abnormal samples. Exemplarily, 1 represents a normal sample and -1 represents an abnormal sample; (2) Data preprocessing: preprocess the data, including missing value processing, standardization, feature engineering, etc., to ensure that the data is ready to be input into the model; (3) Divide the training set and the test set: divide the dataset into a training set and a test set, first use the training set to train the model, and then use the test set to evaluate the model performance; (4) Model selection and creation: select the Isolation Forest as the anomaly detection model. In the machine learning scikit-learn library, the IsolationForest class can be used to create an Isolation Forest model; (5) Model training: use the training set to train the Isolation Forest model; (6) Model evaluation: use the test set to evaluate the performance of the model, and some metrics such as accuracy, precision, recall, F1-score, etc. can be used to measure the performance of the model.

[0091] Use the trained Isolation Forest anomaly detection model to perform anomaly detection on the dimensionality-reduced labeled dataset to obtain an anomaly score or an anomaly label. If an anomaly score is obtained, the higher the anomaly score, the higher the likelihood of the label of the labeled dataset being abnormal. In addition, the anomaly points detected by the Isolation Forest anomaly detection model can also be analyzed, and the model parameters or the data preprocessing steps can be continuously adjusted to improve the model performance.

[0092] The core idea of the Isolation Forest is to perform splitting by randomly selecting a feature and randomly selecting a threshold on that feature. For image data, this splitting operation may not be able to fully capture the complex structures and semantic information in the image, especially when there are complex associations between anomaly points and normal points in multiple dimensions. For image data, the basic splitting operation of the Isolation Forest may indeed be insufficient to capture the complex structures and semantic information. When dealing with this problem, multi-scale information can be introduced into the Isolation Forest to make the image contain information at multiple scales, thereby improving the sensitivity of the model to abnormal associations in different dimensions.

[0093] Therefore, when training the Isolation Forest anomaly detection model, the original training dataset after dimensionality reduction and with labeled information can be obtained, and the original training dataset can be processed using a multi-scale expression method to obtain a multi-scale training dataset; the Isolation Forest anomaly detection model can be iteratively trained through the multi-scale training dataset to obtain the trained Isolation Forest anomaly detection model.

[0094] Among them, processing the original training dataset using a multi-scale representation means: processing each original training data in the original training dataset using a multi-scale representation to obtain the multi-scale information of each original training dataset. The set of the multi-scale information of all original training datasets is the preprocessed dataset, that is, the preprocessed training dataset includes the multi-scale information of the original training dataset. Specifically, the steps of processing the original training dataset using a multi-scale representation to obtain a multi-scale training dataset include: using the image pyramid of the original training dataset as the multi-scale training dataset; or, sliding several windows of different scales on the original training dataset and capturing local information at different scales to obtain the multi-scale training dataset; or, extracting features from the original training dataset through a multi-scale feature extractor to obtain the multi-scale training dataset; or, performing several decompositions of different scales on the original training dataset to obtain several subsets of data, and using the set of the several subsets of data as the multi-scale training dataset. Among them, by integrating the detection results obtained by processing the several subsets of data through an isolation forest anomaly detection model, the anomaly detection result of the training dataset can be obtained; or, using the versions of the original training dataset at several different resolutions as the multi-scale training dataset; or, performing several decompositions of different scales on the original training dataset to obtain several subsets of data, making any combination of the subsets of data at different scales to obtain a combined dataset, and using the set of all combined datasets obtained by combination as the multi-scale training dataset.

[0095] Considering multi-scale information is to capture features in the image more comprehensively because images usually contain information at different scales. Introducing multi-scale information can make the isolation forest model more adaptable and more sensitive to abnormal associations at different scales. The present application provides the following implementation methods for multi-scale information:

[0096] (1) Pyramid scale: Image pyramids are a common method for implementing multi-scale. It refers to images being scaled down or up to different scales. Isolation forests can be trained separately at these scales and their results integrated; smaller scales may be more likely to capture local details, while larger scales are more suitable for capturing overall structures. (2) Sliding window: Apply the isolation forest to the sliding window of the image. Each window represents a different scale; by sliding windows of different sizes over the image, local information at different scales can be captured. (3) Multi-scale feature extraction: Use a multi-scale feature extractor, such as using convolutional kernels of different sizes or feature maps at different levels in deep learning. This can obtain feature representations with different scale information, which can then be used for the training of the isolation forest. (4) Decomposition and integration: Decompose the image into sub-images of different scales, apply the isolation forest for anomaly detection separately, and then integrate their results. Exemplarily, integration can be achieved by weighted averaging or other aggregation methods of the anomaly scores of each sub-image. (5) Multi-resolution image processing: When processing images, a multi-resolution method can be used. By retaining image information at different resolutions and selectively inputting it into the isolation forest, this can be achieved through pixel-level pyramid operations or multi-scale representations of the image. (6) Feature combination: Combine the features extracted at different scales to form a more representative feature representation. This can be done through feature fusion methods, such as concatenation, stitching, or weighted summation.

[0097] By introducing multi-scale information through any of the above implementation methods for multi-scale information, the spatial distribution, structure, and key information at specific scales of the image can be more comprehensively considered, resulting in a significant improvement in the isolation forest anomaly detection model.

[0098] After introducing the multi-scale information of the picture and processing it with the isolation forest anomaly detection model, the scores of each labeled data as shown in Figure 6 can be obtained. By analyzing the anomaly scores, outliers can be identified by setting a threshold. The lower the anomaly score, the more likely the sample is an outlier. The anomaly scores of most samples are concentrated in the higher value region, and these samples are considered normal. In the region with lower anomaly scores, that is, the left tail, there are some samples concentrated, and these samples are marked as potentially outliers by the isolation forest. By adjusting the parameters of the isolation forest, such as contamination, the identification of outliers can be changed. A higher contamination may result in more samples being marked as outliers.

[0099] In a preferred embodiment, after the anomaly detection of the labeled dataset is completed, the features of the dataset can also be extracted to evaluate the annotation quality, including the evaluation of accuracy and consistency, so as to determine the quality level of the annotation. For an exemplary illustration, the consistency of the labeled dataset can be evaluated by calculating the Cohen's Kappa coefficient. Among them, the Cohen's Kappa coefficient is a consistency index for measuring the performance of a classifier, which is used to measure the consistency between the clustering result and the true label, and is used to evaluate the consistency of the average measurement results of all observers in multiple measurements. The Cohen's Kappa coefficient takes into account the accuracy of the classifier and corrects the influence of random classification. Its value ranges between -1 and 1. A value of 1 indicates perfect consistency, 0 indicates consistency with a random classifier, and -1 indicates complete inconsistency.

[0100] Taking the aforementioned iris dataset as an example, when evaluating its consistency, a clustering algorithm such as the K-means algorithm can be used to group the data and compare it with the true label to evaluate the consistency of the clustering result. Further, the consistency of the clustering can also be evaluated through a confusion matrix and the Cohen's Kappa coefficient.

[0101] Among them, the confusion matrix is an N×N matrix, where N represents the number of categories, as Figure 7 shown, which shows the correspondence between the clustering result and the true label. The elements on the diagonal represent the number of samples correctly classified, and the elements off the diagonal represent the number of samples misclassified. Each category has its own index in the rows and columns of the confusion matrix, and the accuracy and misclassification situation of the clustering can be evaluated through this matrix.

[0102] Preferably, the anomaly detection method can also be optimized and adjusted, including improving the anomaly detection algorithm, adjusting the PCA algorithm parameters or other feature engineering methods, to improve the accuracy and reliability of the anomaly detection results. For an exemplary illustration, the optimization and adjustment can be carried out through the following steps:

[0103] (1) Initial parameter setting: Run the algorithm with default parameters to obtain the initial results, and use the good initial results as the benchmark for further optimization. (2) Cross-validation: Use cross-validation techniques to divide the dataset into a training set and a validation set. Train the model on the training set and evaluate the model performance on the validation set. This helps to avoid overfitting and provides a more reliable performance evaluation. (3) Parameter tuning: Adjust the number of components in principal component analysis and try different numbers of principal components to see if there is a better dimensionality reduction effect. Adjust the parameters of the isolation forest, such as increasing or decreasing the number of trees and adjusting the depth of the trees. Use techniques such as grid search or random search to find the best combination of parameters. (4) Performance evaluation: Use the validation set to evaluate the performance of the adjusted model and compare the performance metrics under different parameter settings, such as accuracy, recall, F1-score, etc. (5) Strategies such as adaptive learning rate and dynamic adjustment can be adopted to more flexibly adapt to data changes. (6) Model interpretability: Consider the interpretability of the model, especially for the isolation forest algorithm. Understand which features play a key role in anomaly detection and need to interpret the model output based on domain knowledge. (7) Robustness testing: Test on different datasets to ensure the robustness of the model. Test for different types of data and annotation situations to understand the adaptability of the model. (8) Documentation: Record the results of each parameter adjustment and performance metrics and provide detailed documentation so that others can understand and reproduce the method optimization process. (9) Final evaluation: Use an independent test set for the final evaluation to ensure that the model performs well on unprocessed data.

[0104] The above anomaly detection method for labeled data obtains the labeled dataset to be evaluated; performs dimensionality reduction processing on the labeled dataset to be evaluated to obtain the dimensionality-reduced labeled dataset; inputs the dimensionality-reduced labeled dataset into the trained isolation forest anomaly detection model for processing to obtain the corresponding anomaly detection results, thereby overcoming the problem of data dimensionality disaster to a certain extent, reducing the computational complexity, and improving the anomaly detection efficiency for high-dimensional labeled data.

[0105] It should be understood that although Figure 3 、 Figure 5 the steps in the flowchart of Figure 3 、 Figure 5At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily need to be executed and completed at the same moment, but can be executed at different moments. The execution order of these sub-steps or stages does not necessarily need to be sequential, but can be executed alternately or in turns with at least some of the sub-steps or stages of other steps or other steps.

[0106] In one embodiment, as Figure 8 shown, a dimensionality reduction device for labeled data is provided, including a feature extraction unit, a dimensionality reduction unit, and a fusion unit, where:

[0107] The feature extraction unit is configured to input the labeled data set to be evaluated into a convolutional neural network, and perform feature extraction and downsampling processing respectively to obtain corresponding channel maps and original feature maps;

[0108] The dimensionality reduction unit is configured to perform dimensionality reduction processing on the pixel values in the channel map by using incremental principal component analysis with a regularization term introduced, to obtain a feature map regarding the covariance matrix;

[0109] The fusion unit is configured to fuse the feature map regarding the covariance matrix with the original feature map to obtain the dimensionality-reduced labeled data set.

[0110] In a specific implementation manner, the dimensionality reduction unit is configured to perform dimensionality reduction processing on the pixel values in the channel map by using incremental principal component analysis with a regularization term introduced, to obtain a feature map regarding the covariance matrix, specifically including: dividing the pixel values in the channel map into several batches of pixel values, and obtaining an optimized objective function after introducing the regularization term; according to the optimized objective function, calculating the principal components and eigenvalues of each batch of pixel values one by one; accumulating the principal components and eigenvalues calculated for each batch of pixel values to obtain corresponding final principal components and final eigenvalues; projecting the pixel values in the channel map onto the final principal components to obtain the feature map regarding the covariance matrix.

[0111] In a specific implementation manner, the dimensionality reduction unit is configured to obtain an optimized objective function after introducing the regularization term, specifically including: obtaining an initial objective function, where the initial objective function represents obtaining a projection matrix that maximizes the variance of the pixel values in the channel map after projection; obtaining a preset regularization parameter and a regularization term, where the regularization term includes the sum of the squares of all elements in the projection matrix; adjusting the initial objective function according to the regularization parameter and the regularization term to obtain the optimized objective function.

[0112] In a specific implementation manner, the dimensionality reduction unit is used to calculate the principal components and eigenvalues of each of the batch pixel values one by one according to the optimization objective function, which specifically includes: solving the optimization objective function based on the first batch pixel value among the several batch pixel values to obtain the pixel variance of the first batch pixel value; calculating the pixel mean of the first batch pixel value, and calculating the covariance matrix of the first batch pixel value according to the pixel variance and the pixel mean; performing eigen decomposition on the covariance matrix to obtain the principal components and eigenvalues corresponding to the first batch pixel value; and updating the principal components and eigenvalues of the remaining batch pixel values one by one through the principal components and eigenvalues corresponding to the first batch pixel value.

[0113] In one embodiment, as Figure 9 shown, an outlier detection system for labeled data is provided, including: a data set acquisition module, a dimensionality reduction device for labeled data as described in the foregoing embodiment, and a data set outlier detection module, where:

[0114] The data set acquisition module is used to acquire a labeled data set to be evaluated;

[0115] The dimensionality reduction device for labeled data as described in the foregoing embodiment is used to perform dimensionality reduction processing on the labeled data set to be evaluated to obtain a dimensionality-reduced labeled data set;

[0116] The data set outlier detection module is used to input the dimensionality-reduced labeled data set into a trained isolation forest outlier detection model for processing to obtain a corresponding outlier detection result.

[0117] In a specific implementation manner, as Figure 10 shown, the system further includes a model training module, which is used to: acquire an original training data set with dimensionality reduction and labeled information, process the original training data set by using a multi-scale expression method to obtain a multi-scale training data set; and perform iterative training on the isolation forest outlier detection model through the multi-scale training data set to obtain the trained isolation forest outlier detection model.

[0118] In a specific embodiment, the model training module is used to process the original training dataset by using a multi-scale expression method to obtain a multi-scale training dataset, specifically including: using the image pyramid of the original training dataset as the multi-scale training dataset; or, sliding several windows with different scales on the original training dataset and capturing local information at different scales to obtain the multi-scale training dataset; or, extracting features from the original training dataset through a multi-scale feature extractor to obtain the multi-scale training dataset; or, performing several decompositions of the original training dataset at different scales to obtain several sub-datasets, and using the set of the several sub-datasets as the multi-scale training dataset, wherein, integrating the detection results obtained by processing the several sub-datasets through an isolation forest anomaly detection model to obtain the anomaly detection result of the training dataset; or, using the versions of the original training dataset at several different resolutions as the multi-scale training dataset; or, performing several decompositions of the original training dataset at different scales to obtain several sub-datasets, arbitrarily combining the sub-datasets at different scales to obtain a combined dataset, and using the set of all the combined datasets obtained by combination as the multi-scale training dataset.

[0119] For the specific limitations of the dimensionality reduction device for labeled data, reference can be made to the limitations of the dimensionality reduction method for labeled data in the above text, which will not be elaborated here; for the specific limitations of the anomaly detection system for labeled data, reference can be made to the limitations of the anomaly detection method for labeled data in the above text, which will not be elaborated here. Each module in the above-mentioned dimensionality reduction device for labeled data and the anomaly detection system for labeled data can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor in the computer device in the form of hardware or be independent of it, or can be stored in the memory in the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above-mentioned modules.

[0120] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 11 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the labeled dataset to be evaluated, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a dimensionality reduction method for labeled data.

[0121] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structural diagram may be as Figure 11 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the labeled data set to be evaluated, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for detecting anomalies in labeled data.

[0122] Those skilled in the art can understand that Figure 11 the structure shown in

[0123] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0124] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the dimensionality reduction method for labeled data as described in the foregoing embodiment.

[0125] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0126] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0127] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An abnormal detection method for labeled data, characterized in that The method includes: Obtaining an annotated data set to be evaluated; Inputting the annotated data set to be evaluated into a convolutional neural network, and performing feature extraction and downsampling processing respectively to obtain corresponding channel maps and original feature maps; Dividing the pixel values in the channel map into several batches of pixel values, and obtaining an optimized objective function after introducing a regularization term; according to the optimized objective function, calculating the principal components and eigenvalues of each batch of pixel values one by one; accumulating the principal components and eigenvalues calculated for each batch of pixel values to obtain corresponding final principal components and final eigenvalues; projecting the pixel values in the channel map onto the final principal components to obtain a feature map of the covariance matrix; Fusing the feature map of the covariance matrix with the original feature map to obtain a dimension-reduced annotated data set; Obtaining an original training data set with dimension reduction and annotation information, and processing the original training data set using a multi-scale expression method to obtain a multi-scale training data set; Iteratively training an isolation forest anomaly detection model through the multi-scale training data set to obtain a trained isolation forest anomaly detection model; Inputting the dimension-reduced annotated data set into the trained isolation forest anomaly detection model for processing to obtain corresponding anomaly detection results.

2. The abnormal detection method for labeled data according to claim 1, characterized in that, The steps of obtaining the optimized objective function after introducing a regularization term include: Obtaining an initial objective function, where the initial objective function represents obtaining a projection matrix that maximizes the variance of the pixel values in the channel map after projection; Obtaining a preset regularization parameter and a regularization term, where the regularization term includes the sum of the squares of all elements in the projection matrix; Adjusting the initial objective function according to the regularization parameter and the regularization term to obtain an optimized objective function.

3. The abnormal detection method for labeled data according to claim 1, characterized in that, The steps of calculating the principal components and eigenvalues of each batch of pixel values one by one according to the optimized objective function include: Solving the optimized objective function based on the first batch of pixel values among the several batches of pixel values to obtain the pixel variance of the first batch of pixel values; Calculating the pixel mean of the first batch of pixel values, and calculating the covariance matrix of the first batch of pixel values according to the pixel variance and the pixel mean; Performing eigenvalue decomposition on the covariance matrix to obtain the principal components and eigenvalues corresponding to the first batch of pixel values; Updating the principal components and eigenvalues of the remaining batches of pixel values one by one through the principal components and eigenvalues corresponding to the first batch of pixel values.

4. The abnormal detection method for labeled data according to claim 1, characterized in that The steps of processing the original training data set using a multi-scale expression method to obtain a multi-scale training data set include: Using the image pyramid of the original training data set as the multi-scale training data set; or, Sliding several windows with different scales on the original training data set and capturing local information at different scales to obtain the multi-scale training data set; or, Performing feature extraction on the original training data set through a multi-scale feature extractor to obtain the multi-scale training data set; or, Performing several different-scale decompositions on the original training dataset to obtain several sub-datasets, and using the set of the several sub-datasets as the multi-scale training dataset; or, Using the versions of the original training dataset at several different resolutions as the multi-scale training dataset; or, Performing several different-scale decompositions on the original training dataset to obtain several sub-datasets, making arbitrary combinations of the sub-datasets at different scales to obtain combined datasets, and using the set of all the combined datasets obtained by combination as the multi-scale training dataset.

5. A quality inspection system for labeled data, characterized in that, The system includes: A dataset acquisition module for acquiring an annotated dataset to be evaluated; A feature extraction unit for inputting the annotated dataset to be evaluated into a convolutional neural network to perform feature extraction and downsampling processing respectively, so as to obtain corresponding channel maps and original feature maps; A dimensionality reduction unit for: dividing the pixel values in the channel map into several batches of pixel values, and obtaining an optimized objective function with a regularization term introduced; calculating the principal components and eigenvalues of each of the batches of pixel values one by one according to the optimized objective function; accumulating the principal components and eigenvalues calculated for each of the batches of pixel values to obtain corresponding final principal components and final eigenvalues; projecting the pixel values in the channel map onto the final principal components to obtain a feature map of the covariance matrix; A fusion unit for fusing the feature map of the covariance matrix with the original feature map to obtain a dimensionality-reduced annotated dataset; A model training module for: acquiring the original training dataset with dimensionality reduction and annotation information, processing the original training dataset using a multi-scale expression to obtain a multi-scale training dataset; iteratively training an isolation forest anomaly detection model through the multi-scale training dataset to obtain a trained isolation forest anomaly detection model; A dataset anomaly detection module for inputting the dimensionality-reduced annotated dataset into the trained isolation forest anomaly detection model for processing to obtain corresponding anomaly detection results.

6. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the anomaly detection method for annotated data according to any one of claims 1 to 4.

Images