Image anomaly detection method by aligning multi-spectral cross-domain features in autonomous driving
By aligning the characteristics of the source domain and the target domain in autonomous driving through multi-spectral cross-domain feature detection method, the problem of data scarcity is solved, more accurate image abnormality detection is achieved, and the safety and detection performance of autonomous driving is improved.
Patent Information
- Application Number
- CN202110955581.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-08-19
AI Technical Summary
The detection of cross-domain image abnormality in autonomous driving faces the problem of data scarcity, and the prior art is difficult to effectively detect image abnormalities, affecting safety.
Using multi-spectral cross-domain feature detection method, the features of the source domain and the target domain are aligned through the neural network model, and the multi-spectral feature fusion module, feature extractor, category discriminator and domain discriminator are used to calculate the center point and loss function to achieve exceptions. Test.
Achieve better transfer learning effect of abnormal detection under the condition of few samples, improving the accuracy and universality of detection, and significantly improving AUROC performance.
Smart Images

Figure CN113792609B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a method for detecting image anomalies by aligning multi-spectral cross-domain features in autonomous driving. Background Art
[0002] At present, autonomous driving vehicles are generally equipped with visual image acquisition systems. The visual image acquisition system uses a camera to convert the detected target into an image signal, which is transmitted to a dedicated image processing system. According to the pixel distribution, brightness, color and other information, it is converted into a digital signal. The image processing system then performs various operations on these signals to extract the characteristics of the target and then detect the corresponding target. However, the accuracy of target detection depends on high-quality image acquisition. When the captured image is overexposed, obscured by smoke, or obscured by rain, it will have a great impact on the accuracy of target detection, which directly affects the safety of autonomous driving.
[0003] Cross-domain image anomaly detection transfer learning is a hot research direction in machine learning. Most cross-domain image anomaly detection research is mainly conducted under the training set with sufficient existing data. In the scenario of autonomous driving, a major problem in cross-domain image anomaly detection research is that it is difficult to obtain a large amount of target domain data in real scenes. Because most of the images collected by vehicles during daily driving are normal, and it is difficult for real vehicles to continuously obtain abnormal image data on the road. At the same time, because autonomous driving vehicles have to face many road environments during actual driving, it is extremely difficult to obtain training data in all road environments.
[0004] In the prior art, Chinese invention patent application number CN201910251114.6 discloses a retinal abnormality detection method, device, equipment and storage medium, the method comprising: obtaining an original retinal image of a retina; determining the image quality of the original retinal image; when the image quality meets the preset quality conditions, performing image style transfer preprocessing on the original retinal image to obtain a target retinal image; and determining retinal abnormality information based on the target retinal image. It is achieved that before retinal abnormality detection, the image quality map of the retinal image is determined and the retinal image is preprocessed with style transfer, so that the retinal image used to determine retinal abnormalities is a high-quality and style-consistent image, which solves the problem of inaccurate retinal abnormality detection caused by poor retinal image quality and style differences, and can extract sufficient accurate image features from high-quality and style-consistent target retinal images for retinal abnormality detection, thereby improving the accuracy of retinal abnormality detection. However, a method for detecting image abnormalities by aligning multi-spectral cross-domain features in autonomous driving has not been proposed.
[0005] For example, Chinese invention patent application number CN201710749768.2 discloses an image anomaly detection method combining CNN transfer learning and SVDD. The image around the image object to be detected is manually intercepted according to the video data, and the pillar number data set to be detected is prepared. The deep features of the image data are expressed by the convolutional neural network, and the network model with pre-trained weights and parameters is used. The features of the pillar number samples are fully extracted through the trained network model to solve the problem of minority data in unbalanced data; a positive sample feature set that needs to be trained in the classifier is constructed, and finally the support vector data description algorithm and grid search are used to optimize the parameters to form a normal domain for positive sample feature training, and the contact network number status is recognized through this boundary. The invention has a high level of automation processing, which can greatly reduce the workload of operators, detect abnormal problems of contact network pillar numbers early, and improve inspection efficiency. However, a method for detecting image anomalies by aligning multi-spectral cross-domain features in autonomous driving is also not proposed.
[0006] In actual autonomous driving, it is necessary to detect anomalies in the collected image data, so as to take appropriate measures in time for abnormal images to avoid affecting the safety of autonomous driving. In addition, it is also necessary to use the domain adaptation method (DA) of transfer learning to help use a small amount of target domain data to achieve feature alignment with the larger source domain data, so as to meet the anomaly detection needs in the target domain. Summary of the invention
[0007] The present invention aims at the problem of cross-domain anomaly detection in difficult scenarios that actually exist in autonomous driving, and proposes a method for detecting image anomalies by aligning multi-spectral cross-domain features in autonomous driving.
[0008] The method for detecting image abnormality of the present invention comprises the following steps:
[0009] Step 1, constructing a neural network based on initializing the feature extractor G, the multi-spectral feature fusion module M, the classifier C and the domain discriminator D;
[0010] Initialize the parameters of the multi-spectral feature fusion module, feature extractor, category discriminator, and domain discriminator of the neural network model;
[0011] Step 2: Calculate the source domain center point μ s and the center point μ of the target domain t ;
[0012] Step 3: Sample source domain and target domain samples, input them into the feature generator G and obtain the corresponding feature representation output;
[0013] Step 4: Calculate the category loss between the source domain sample model prediction and the target domain sample model prediction output and the true label
[0014] Step 5: Calculate the domain adversarial loss between the predicted output and the true label of the source domain sample model and the target domain sample model for this batch
[0015] Step 6: Calculate the distance loss between the predicted output of the source domain sample model and the target domain sample model and the samples with the true label for this batch
[0016] Step 7: Calculate the overall loss of the entire model
[0017] Step 8: Use stochastic gradient descent (SGD) to update the model and align the features of source domain samples and target domain samples.
[0018] Step 9, repeat steps 2 to 8 until the number of iterations reaches the maximum;
[0019] Step 10, in the testing phase, calculate the feature center μ of all normal instances;
[0020] Step 11: Calculate the test sample Anomaly score
[0021] Furthermore, the step 1 of constructing a neural network comprises the following steps:
[0022] Step 1.1, use the ResNet-50 network model pre-trained on ImageNet as the feature extractor G, and delete the global average pooling layer of ResNet-50;
[0023] Step 1.2, use a fully connected layer to reduce the size of the feature extractor G to 256 dimensions;
[0024] Step 1.3, design a multi-spectral feature fusion module M to integrate information from different spectral components, and connect the category discriminator, domain discriminator D, and center clustering module after the feature extractor G respectively.
[0025] Furthermore, the calculation of the source domain center point μ in step 2 is s and the center point μ of the target domain t , including the following steps:
[0026] Step 2.1: Based on the training data set, the source domain normal sample feature data after the multi-spectral feature fusion module M is input into the source domain normal sample feature center point calculation model to obtain the source domain normal sample feature center point μ s, the calculation model of the feature center point of the normal sample in the source domain is as follows (1):
[0027]
[0028] In the above formula (1): μ s is the feature center point of the source domain, n s is the number of normal samples in the source domain involved in training, represents the i-th training sample in the source domain, is the feature of the i-th instance of the current source domain calculated by the feature extractor G, It represents the output features of the normal samples in the source domain after the multi-spectral feature fusion module combines the information from different spectral components;
[0029] Step 2.2: Input the target domain normal sample data into the target domain feature center point calculation model to obtain the target domain normal sample feature center point μ t , the calculation model of the center point of the normal sample in the target domain is as follows (2):
[0030]
[0031] In the above formula (2): μ t is the feature center point of the target domain, n t is the number of normal samples involved in training in the target domain, represents the jth training sample in the target domain, is the feature of the jth instance of the target domain calculated by the feature extractor G, It represents the output features of normal samples in the target domain after the multi-spectral feature fusion module combines information from different spectral components.
[0032] Furthermore, the step 3 of obtaining the feature representation output of the source domain sample and the target domain sample includes the following steps:
[0033] Step 3.1: Randomly sample source domain samples of small batches of size B (Batch size) With the target domain sample
[0034] Step 3.2, and the source domain samples and target domain samples Input into the feature generator G to obtain the corresponding feature representation;
[0035] In step 3.3, the generated feature representation is input into the category discriminator C, the domain discriminator D and the center clustering module respectively, and then the corresponding output is obtained, in which the number of target domain samples is much smaller than the number of source domain samples.
[0036] Furthermore, step 4 calculates the category loss between the source domain sample model prediction and the target domain sample model prediction output and the true label
[0037] In step 2, the source domain samples and target domain samples Under the supervision of the annotation information, the dual-domain joint cross entropy loss function is used to calculate the category loss between the source domain sample model prediction and the target domain sample model prediction output and the true label of this batch. As shown in the following formula (3):
[0038]
[0039] In the above formula (3): n s is the number of normal samples in the source domain participating in training, n t is the number of normal samples involved in training in the target domain, It represents the output features of the normal samples in the source domain that are combined with information from different spectral components after passing through the multi-spectral feature fusion module. It represents the output features of the target domain normal samples after the multi-spectral feature fusion module combines the information from different spectral components, and C is the category classifier.
[0040] Furthermore, in step 5, under the supervision of the label information of the source domain samples and the target domain samples in step 4, the domain adversarial loss function is used to calculate the domain adversarial loss between the predicted output of the source domain sample model and the target domain sample model of this batch and the true label. As shown in formula (4):
[0041]
[0042] In the above formula (4): It represents the expected value of the loss calculated by the discriminator on the output of the source domain data of the current batch input.
[0043] Furthermore, in step 6, the distance loss between the predicted output of the source domain sample model and the target domain sample model and the sample of the true label is calculated. As shown in formula (5):
[0044]
[0045] In the above formula (5): ||·|| 2 represents L2 regularization, μ s and μ t are the center points of the source domain and the target domain respectively.
[0046] Furthermore, the overall loss of the entire model is calculated as described in step 7 As shown in formula (6),
[0047]
[0048] In the above formula (6), α and β are two parameters used to balance the loss.
[0049] Furthermore, in step 8, stochastic gradient descent SGD is used to update the model to achieve alignment of source domain samples and target domain samples:
[0050] The overall loss of the entire model obtained according to step 7 The Stochastic Gradient Descent (SGD) algorithm is used for back propagation to update the feature generator G, the category discriminator C, and the substituted parameters to minimize the loss function, so that the feature generator G generates a more effective feature representation and achieves the alignment of the source domain samples and the target domain samples:
[0051] Furthermore, the feature center μ of all normal instances is calculated as described in step 10, as shown in the following formula (7):
[0052]
[0053] In the above formula (7), n s is the number of instances in the source domain that participate in training, is the feature of the i-th instance of the source domain in the current batch calculated by the feature extractor G, It means that the output features from different spectral component information are integrated after passing through the multi-spectral feature fusion module.
[0054] Furthermore, the test sample is calculated in step 11 Anomaly score As shown in formula (8):
[0055]
[0056] In the above formula (8), is the feature of the mth instance of the test set in the current batch calculated by the feature extractor G, It represents the output features from different spectral components after the multi-spectral feature fusion module, and μ represents the feature center of the normal instance.
[0057] Compared with the existing similar methods, the method for detecting image anomalies of the present invention has the following advantages:
[0058] 1. Most of the existing anomaly detection methods for autonomous driving transfer learning require sufficient labeled samples for training, which is difficult to achieve in the actual situation of autonomous driving. The present invention studies a more difficult and practical anomaly detection scenario. In this scenario, there is no abnormal data involved in the model training phase, and the normal data in the target domain is limited during the training phase. The assumptions in such a scenario are closer to the actual situation.
[0059] 2. The present invention proposes a method of Multi-spectral Cross-domain Representation Alignment (MsRA). MsRA first generalizes the deep feature map to the spectrum, and then constructs a multi-spectral feature representation by weighted combination of different spectral components, which makes up for the problem of insufficient information caused by limited target training data. This enables the method of the present invention to achieve better anomaly detection transfer learning effect under the condition of few samples.
[0060] 3. The existing anomaly detection methods for autonomous driving transfer learning basically still rely on traditional anomaly detection methods to detect abnormal samples, which makes the overall model obviously include two separate steps of feature extraction and anomaly detection. The method for detecting image anomalies described in the present invention uses adversarial loss and center matching loss to learn domain invariant features and class compact features respectively, and uses the distance from the sample instance to the feature center point as the anomaly score without training an additional anomaly detection model. The method for detecting image anomalies described in the present invention uses an end-to-end approach in both training and testing stages, and does not need to perform feature extraction first, and then rely on training other existing anomaly detection methods to detect abnormal samples.
[0061] 4. The method for detecting image anomalies described in the present invention is highly universal and can adapt to a variety of transfer learning anomaly detection tasks. It is not limited to a specific task. However, various transfer learning autonomous driving anomaly detection methods in the prior art are basically aimed at a specific task and have great limitations.
[0062] 5 The method for detecting image anomalies described in the present invention has been extensively experimented on a cross-domain benchmark test set. The final results show the effectiveness of the proposed MsRA method, which has a significant improvement in the model migration performance compared with other existing advanced AUROC methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 A general overview of the research motivation for the method for detecting image anomalies according to the present invention;
[0064] Figure 2The overall algorithm framework diagram of the method for detecting image anomalies by aligning multi-spectral cross-domain features according to the present invention;
[0065] Figure 3 is the AUROC (%) value for anomaly detection on the Calculator class (ClipArt→Product) task of the OfficeHome dataset when each spectral component is used separately.
[0066] Figure 4 This is a hyperparameter sensitivity analysis diagram of the MsRA algorithm on the OfficeHome dataset.
[0067] Figure 5 The present invention is a flowchart of the method for detecting anomalies in images by aligning multi-spectral cross-domain features. DETAILED DESCRIPTION
[0068] The following is attached to the instruction manual Figure 1-5 The specific embodiment of the method for detecting image abnormality of the present invention is further described in detail.
[0069] In order to better understand the method for detecting image abnormalities according to the present invention, the following briefly introduces the structures associated with the method for detecting image abnormalities according to the present invention, for example, the principle and application of the feature extractor G, the multi-spectral feature fusion module M, the classifier C and the domain discriminator D.
[0070] In the field of autonomous driving image anomaly detection, the anomaly detection problem in the scenario setting where only source domain normal data and a limited number of target domain normal data are involved in the training phase aims to establish an effective anomaly score function to distinguish normal and abnormal instances in the target area. Due to domain drift and shortage of target training data, the anomaly score function constructed based only on source domain data or target data usually performs poorly. In order to solve this problem, the method for detecting image anomalies of the present invention comprises a feature extractor G, a multi-spectral feature fusion module M, The classifier C and the domain discriminator D are used to align the source domain and the target domain. At the same time, the method for detecting image anomalies in the present invention constructs an anomaly score function based on the feature representation of the alignment of the two domain features, that is, the deep features extracted by the feature extractor are first generalized to the spectrum, and then a multi-spectral feature representation is constructed by fusing different spectral components, thereby introducing more information about the input. In the multi-spectral feature space, the feature extractor is forced to learn domain-invariant features through adversarial loss and center clustering loss, such as Figure 2 When the source domain and the target domain are aligned, the data of the source domain and the target domain can be combined to calculate the center point of the normal class, and the distance from the instance to the center point is defined as the anomaly score. In the training phase, n s Indicates that from the source domain Xs ×Y s The sampled independent and identically distributed normal instances, i.e. Use n t Indicates that from the target domain X t ×Y t The sampled independent and identically distributed normal instances, i.e. And set n t <<n s The test data set of the target domain contains normal instances and abnormal instances, denoted as Where n test is the number of test samples, The source training data are Target training data and target test data The corresponding label of the target domain data is 0, which indicates normal and 1 indicates abnormal samples. s Sharing the same label space and sampled from different data distributions, there is domain drift between the source and target domains, and the goal is to establish a well-performing anomaly score function for the target domain. Where s is the anomaly score.
[0071] In the method for detecting image anomalies of the present invention, the deep convolutional neural network has excellent feature extraction capability, which is used for feature extractor G to extract features. After obtaining the deep feature map, the deep CNN-based methods in the prior art all use global average pooling (GAP) to process the extracted deep feature map for downstream tasks, while GAP actually only uses the average value information, which is insufficient to capture rich input pattern information. Through GAP spectrum analysis, it is proved that GAP actually only uses the information in the lowest spectral component.
[0072] First, the Discrete Cosine Transform (DCT) is defined as follows:
[0073]
[0074] s t i∈{0, 1, …, L-1}
[0075] In the above formula: is an input vector of length L, is the spectrum obtained by DCT of x. i It is the spectral component at position i in spectrum F. The two-dimensional DCT based on DCT is shown as follows:
[0076]
[0077] s·tu∈{O, 1,…,H-1}, v∈{0, 1,…,W-1}
[0078] In the above formula: is an input matrix of height H and width W, and is the two-dimensional DCT corresponding to x 2d Spectrum, is the spectrum F 2d The spectrum component at the (u, v) position, spectrum F 2d The inverse two-dimensional DCT of is expressed as:
[0079]
[0080] sth∈{0, 1,…,H-1},w∈{0,1,…,W-1}
[0081] For the convenience of calculation, the constant normalization coefficient that does not affect the analysis results is ignored in the above formula;
[0082] In particular, considering the case of u = 0 and v = 0, the lowest spectral component of the input x2d It is expressed as:
[0083]
[0084] In the above formula, GAP(·) is the global average pooling. It can be seen that global average pooling is actually a special case of two-dimensional DCT, which is proportional to the lowest spectral component of the input. Due to the very limited target training data, using only global average pooling to process the extracted features will not provide enough information.
[0085] In addition, for simplicity, the DCT weights of the two-dimensional DCT are Expressed as Convert equation (11) into the following equation:
[0086]
[0087] sth∈{0, 1,…,H-1},w∈{0,1,…,W-1}
[0088] Here, it can be observed that an image or feature is actually a combination of different spectral components. This observation helps to consider combining more information from different spectral components of the input to make up for the lack of information due to limited target training data. To this end, the present invention designs the following Figure 2 The multi-spectral feature fusion module M shown, in the multi-spectral feature fusion module:
[0089] make is the feature map extracted by the feature extractor G, where C, H and W are the number of channels, height and width of the feature map respectively. The spectral component of f at the position (u, v) in the spectrum is expressed as follows:
[0090]
[0091] stc∈{O,1,…,C-1}
[0092] In the above formula: f c,:,: is the feature map of the cth channel of f, f c,h,w is the element at position (c, h, w) in f, DCT u,v Indicates the extraction of the spectrum component at the (u, v) position in the spectrum, and the two-dimensional DCT basic function DCT u,v When applied to f, is the corresponding spectral component. Applying multiple two-dimensional DCT basis functions to f will obtain multiple spectral components of f. Assume that K two-dimensional DCT basis functions of size H×W are selected from the DCT basis pool (expressed as ) to construct multi-spectral features as follows:
[0093] For simplicity, the index of the two-dimensional DCT basis function is expressed in one-dimensional format. Considering the different importance of different spectral components, for each spectral component Freq (k) Added learnable importance weights ω k , and merges more information from different spectral components to alleviate the information shortage problem in practical scenarios.
[0094] After constructing the multi-spectral features, domain drift needs to be addressed. Based on the multi-spectral feature representation, exploration is conducted in two directions: domain-invariant feature learning and class-compact feature learning, which are described as follows:
[0095] With the same concept as the Generative Adversarial Network (GAN), many adversarial methods have achieved remarkable success in multi-category classification problems in domain adaptation. Similarly, it is necessary to enforce adversarial training of the feature extractor G, the multi-spectral feature fusion module M, and the domain discriminator D, where the domain discriminator is used to correctly distinguish the source data from the target data, while G and M try to generate features that will confuse the domain recognition results as much as possible. Specifically, the adversarial process is implemented by formula (4) recorded in the specification.
[0096] For simplicity, a gradient reversal layer (GRL) is used and updated by back propagation once in each iteration. When the adversarial training reaches an equilibrium point, the source data and the target data will be aligned in the multi-spectral feature space, so that the domain discriminator cannot distinguish between the source data and the target data. Therefore, domain invariant features are learned through this adversarial training, but only ensuring domain invariance is not enough for anomaly detection. Even if the source domain and the target domain are completely aligned, in some cases, the features of normal instances will still be sparsely distributed in the deep feature space. In this case, classification errors often occur because there is no obvious edge boundary between normal and abnormal categories. To this end, the feature distribution of normal samples is made as dense as possible by minimizing the center clustering loss, as described in formula (5) in the manual.
[0097] By minimizing the center matching loss, the compact classification features of the normal classification can be obtained. In addition, since the features of the normal data are very compact, it is natural to define the distance from the center of the normal class as the abnormal score. The definition of the abnormal score function is as shown in formula (8) in the specification.
[0098] The following is attached to the instruction manual Figure 1-5 The specific embodiment of the method for detecting image abnormality according to the present invention is described in detail. Figure 5 As shown, the steps of the method for detecting image abnormalities of the present invention are as follows:
[0099] Step 1, initializing the parameters of the multi-spectral feature fusion module, feature extractor, category discriminator, and domain discriminator of the neural network model;
[0100] Step 2: Calculate the source domain center point μ s and the center point μ of the target domain t ;
[0101] Step 3: Sample source domain and target domain samples, input them into the feature generator G and obtain the corresponding feature representation output;
[0102] Step 4: Calculate the category loss between the source domain sample model prediction and the target domain sample model prediction output and the true label
[0103] Step 5: Calculate the domain adversarial loss between the predicted output and the true label of the source domain sample model and the target domain sample model for this batch
[0104] Step 6: Calculate the distance loss between the predicted output of the source domain sample model and the target domain sample model and the samples with the true label for this batch
[0105] Step 7: Calculate the overall loss of the entire model
[0106] Step 8: Use stochastic gradient descent (SGD) to update the model and align the features of source domain samples and target domain samples.
[0107] Step 9, repeat steps 2 to 8 until the number of iterations reaches the maximum;
[0108] Step 10, in the testing phase, calculate the feature center μ of all normal instances;
[0109] Step 11: Calculate the test sample Anomaly score
[0110] Furthermore, the step 1 of constructing a neural network comprises the following steps:
[0111] Step 1.1, use the ResNet-50 network model pre-trained on ImageNet as the feature extractor G, and delete the global average pooling layer of ResNet-50;
[0112] Step 1.2, use a fully connected layer to reduce the size of the feature extractor G to 256 dimensions;
[0113] Step 1.3, design a multi-spectral feature fusion module M to integrate information from different spectral components, and connect the category discriminator, domain discriminator D, and center clustering module after the feature extractor G respectively.
[0114] Furthermore, the calculation of the source domain center point μ in step 2 is s and the center point μ of the target domain t , including the following steps:
[0115] Step 2.1: Based on the training data set, the source domain normal sample feature data after the multi-spectral feature fusion module M is input into the source domain normal sample feature center point calculation model to obtain the source domain normal sample feature center point μ s , the calculation model of the feature center point of the normal sample in the source domain is as follows (1):
[0116]
[0117] In the above formula (1): μ s is the feature center point of the source domain, n s is the number of normal samples in the source domain involved in training, represents the i-th training sample in the source domain, is the feature of the i-th instance of the current source domain calculated by the feature extractor G, It represents the output features of the normal samples in the source domain after the multi-spectral feature fusion module combines the information from different spectral components;
[0118] Step 2.2: Input the target domain normal sample data into the target domain feature center point calculation model to obtain the target domain normal sample feature center point μ t , the calculation model of the center point of the normal sample in the target domain is as follows (2):
[0119]
[0120] In the above formula (2): μ t is the feature center point of the target domain, n t is the number of normal samples involved in training in the target domain, represents the jth training sample in the target domain, is the feature of the jth instance of the target domain calculated by the feature extractor G, It represents the output features of normal samples in the target domain after the multi-spectral feature fusion module combines information from different spectral components.
[0121] Furthermore, the step 3 of obtaining the feature representation output of the source domain sample and the target domain sample includes the following steps:
[0122] Step 3.1: Randomly sample source domain samples of small batches of size B (Batch size) With the target domain sample
[0123] Step 3.2, and the source domain samples and target domain samples Input into the feature generator G to obtain the corresponding feature representation;
[0124] In step 3.3, the generated feature representation is input into the category discriminator C, the domain discriminator D and the center clustering module respectively, and then the corresponding output is obtained, in which the number of target domain samples is much smaller than the number of source domain samples.
[0125] Furthermore, step 4 calculates the category loss between the source domain sample model prediction and the target domain sample model prediction output and the true label
[0126] In step 2, the source domain samples and target domain samples Under the supervision of the annotation information, the dual-domain joint cross entropy loss function is used to calculate the category loss between the source domain sample model prediction and the target domain sample model prediction output and the true label of this batch. As shown in the following formula (3):
[0127]
[0128] In the above formula (3): n sis the number of normal samples in the source domain participating in training, n t is the number of normal samples involved in training in the target domain, It represents the output features of the normal samples in the source domain that are combined with information from different spectral components after passing through the multi-spectral feature fusion module. It represents the output features of the target domain normal samples after the multi-spectral feature fusion module combines the information from different spectral components, and C is the category classifier.
[0129] Furthermore, in step 5, under the supervision of the label information of the source domain samples and the target domain samples in step 4, the domain adversarial loss function is used to calculate the domain adversarial loss between the predicted output of the source domain sample model and the target domain sample model of this batch and the true label. As shown in formula (4):
[0130]
[0131] In the above formula (4): It represents the expected value of the loss calculated by the discriminator on the output of the source domain data of the current batch input.
[0132] Furthermore, in step 6, the distance loss between the predicted output of the source domain sample model and the target domain sample model and the sample of the true label is calculated. As shown in formula (5):
[0133]
[0134]
[0135] In the above formula (5): ||·|| 2 represents L2 regularization, μ s and μ t are the center points of the source domain and the target domain respectively.
[0136] Furthermore, the overall loss of the entire model is calculated as described in step 7 As shown in formula (6),
[0137]
[0138] In the above formula (6), α and β are two parameters used to balance the loss.
[0139] Furthermore, in step 8, stochastic gradient descent SGD is used to update the model to achieve alignment of source domain samples and target domain samples:
[0140] The overall loss of the entire model obtained according to step 7 The Stochastic Gradient Descent (SGD) algorithm is used for back propagation to update the feature generator G, the category discriminator C, and the substituted parameters to minimize the loss function, so that the feature generator G generates a more effective feature representation and achieves the alignment of the source domain samples and the target domain samples:
[0141] Furthermore, the feature center μ of all normal instances is calculated as described in step 10, as shown in the following formula (7):
[0142]
[0143] In the above formula (7), n s is the number of instances in the source domain that participate in training, is the feature of the i-th instance of the source domain in the current batch calculated by the feature extractor G, It means that the output features from different spectral component information are integrated after passing through the multi-spectral feature fusion module.
[0144] Furthermore, the test sample is calculated in step 11 Anomaly score As shown in formula (8):
[0145]
[0146] In the above formula (8), is the feature of the mth instance of the test set in the current batch calculated by the feature extractor G, It represents the output features from different spectral components after the multi-spectral feature fusion module, and μ represents the feature center of the normal instance.
[0147] In the experimental stage of the method for detecting image anomalies described in the present invention, in order to compare with other methods, ResNet-50 pre-trained on ImageNet was used as the feature extractor for all CNN-based methods, in which the last layer of ResNet-50 was deleted and the fully connected layer was applied to reduce the feature size to 256 dimensions. For the method of the present invention, the global average pooling of ResNet-50 was further deleted. The classifier is a multilayer perceptron network with 3 fully connected layers. The final prediction output is a one-dimensional vector and is obtained by processing the classifier output using the sigmoid function. The domain classifier is a network with 2 fully connected layers, and its output is a two-dimensional vector. The domain classifier output is processed using the log-softmax function to obtain the domain classification prediction result of the sample. The input image is first resized to 256×256×3 and then cropped to 224×224×3. In addition, since the accessible target samples are limited, the present invention uses data augmentation techniques to enhance the target data, such as flipping and rotation. For network optimization, a learning rate of 1×10 -5 , weight decay is 5×10 -5 and the stochastic gradient descent (SGD) optimizer with momentum 0.9. The batch size of each input was set to 10. The hyperparameters for all experiments were set to α = 1.0, β = 0.1. The model was trained 10 times, each time including 100 epochs, and then the mean and standard deviation of the AUROC value for each task were calculated.
[0148] The data set used in the experiment of the method for detecting image anomalies of the present invention is as follows:
[0149] Office-Home: The Office-Home dataset is a challenging object recognition dataset for domain adaptation. It contains images from 4 different domains: Artistic, Clip Art, Product, and Real-World. Each domain contains 65 categories, and all 4 domains contain the same categories. In addition, the average resolution of the images is 300×300. Here, the "Clip Art" and "Product" domains are selected to build two transfer learning schemes: Clip Art→Product and Product→Clip Art. In addition, for each transfer scheme, only 10 categories are selected from them, which have enough data to create classification tasks, which are listed in the first column of Table 1. Specifically, when the "Bike" class is taken as the normal class, the other 9 categories are regarded as abnormal. All source domain normal samples and a limited number of target domain normal samples are selected to establish the training set. The remaining samples of the target domain normal class and the samples of the other 9 categories of the target domain are used to construct the test dataset. The number of normal target training samples in the target domain is set to 10.
[0150] Office-31: The Office-31 dataset contains 31 object categories from three domains: Amazon, DSLR, and Webcam. The 31 categories in the dataset consist of targets commonly found in office environments, such as keyboards, pens, and bookcases. For the Office-Home dataset, the Amazon and Webcam domains are selected to construct two migration scenarios, Amazon→Webcam and Webcam→Amazon. Ten categories with abundant data are also selected to construct ten single-classification tasks, as shown in the first column of Table 2 listed below. The dataset is constructed in the same way as Office-Home. The number of normal target training samples is 10.
[0151] Digital Set: In computer vision, digital datasets are often used to evaluate model performance. Here, two datasets are used: MINIST and SVHN. MINIST is a large handwritten digital database containing a total of 70,000 grayscale handwritten digital images of size 28×28, where the image is a color image of size 32×32. Both datasets provide digital images from 0 to 9. Two migration schemes are constructed: MNIST→SVHN and SVHN→MNIST. In the experiment, the training set / test set distribution of the original dataset is used. Specifically, MNIST is used as the source domain and the number "0" is used as the normal class. The training set will contain all samples with the class label 0 in the training set of MNIST and a limited number of samples with the class label 0 in the training set of SVHN, while the test dataset consists of all samples in the test set of SVHN. For the digital dataset, the number of target domain training images is set to 50.
[0152] The experiment of the present invention uses the AUROC result as the evaluation criterion: AUROC is the area under the receiver operating characteristic curve (Area Under the Receiver Operating Characteristic curve), which represents the expectation that a uniformly drawn random positive sample ranks before a uniformly drawn random negative sample.
[0153] The following is a detailed evaluation of the performance of the algorithm of the method for detecting image anomalies in the present invention on various migration schemes:
[0154] A.Office-Home
[0155] The anomaly detection results of Clip Art→Product, as shown in the upper part of Table 1 listed below, DSVDD, OCSVM and other standard anomaly detection methods all perform poorly on the transferable anomaly detection task, which indicates that the domain drift between ClipArt and Product hinders the standard anomaly detection method from being extended to the new domain. The results of OCSVM(T, indicating that only the source domain data is used for training) and IF(T) are also inferior to MsRA, which indicates that the limited number of target domain instances is not enough to train a good anomaly detector. In contrast, IRAD has achieved performance improvement through its proposed generative adversarial mechanism, thus proving the effectiveness of making full use of the value of source domain data; in MsRA, by further constructing multi-spectral feature representation by fusing different spectral components, MsRA captures richer input patterns, thereby alleviating the information shortage caused by the limited number of samples in the target domain. The experimental results also show that MsRA obtains more accurate domain alignment with the help of additional information. , MsRA surpasses IRAD on most tasks, thus verifying the effectiveness of MsRA. The lower part of Table 1 shown below shows the experimental results of Product→Clip Art. Conventional anomaly detection methods exhibit a certain degree of domain drift. In addition, it is observed that the IRAD algorithm performs worse than the standard anomaly detection method (i.e., IF(T)) on the “Calculator” class, indicating that complex alignment schemes may fail in some cases. Compared with IRAD, MsRA utilizes a simple network module to align the two domains, thereby achieving stable alignment across different tasks. In addition, MsRA improves the performance of “Calculator” and “Scissors” tasks by 10.4% and 12.3%, respectively.
[0156] Table 1
[0157]
[0158] B.Office-31
[0159] The results of Amazon→Webcam are shown in the upper part of Table 2. It can be seen that BiOST and MsRA achieve better results compared with previous non-transferable anomaly detection methods. These results suggest that in practical transfer learning schemes, the gap between domains should be narrowed to enhance detection capabilities. In addition, compared with other similar methods,
[0160]
[0161] MsRA consistently achieves significant performance gains compared to other approaches. In particular, MsRA
[0162] Table 2
[0163] The AUROC value on the class reaches 97.5%. These significant improvements show that MsRA can stably promote the model performance of the target domain under various transfer schemes. The lower part of Table 2 shows the results of Webcam→Amazon. We can still conclude that MsRA can show the best performance on each task.
[0164] C.Digits
[0165] The top half of Table 3 shows the results for MNIST→SVHN. Due to the migration from grayscale to color images, the model faces more complex input patterns but has access to only a limited number of target samples. Therefore, all methods fail to achieve high AUROC values. In this difficult transfer case, MsRA still shows superior results among the compared methods, demonstrating the effectiveness of the multi-spectral feature fusion strategy, which can enrich the input pattern by fusing different spectral components.
[0166] The lower part of Table 3 shows the results of SVHN→MNIST. The model can more easily transfer knowledge from domains with complex input patterns to domains with simple input patterns. MsRA achieves higher performance improvement compared with other methods. This shows that when more training data with complex input patterns are available, the multi-spectral feature fusion module can bring more improvement to the overall performance.
[0167] Table 3
[0168]
[0169] Table 4 shows the results of the ablation experiments. In order to build a cross-domain anomaly detection model, MsRA unifies the following three key parts: 1) clustering normal data; 2) multi-spectral feature fusion module M to make up for the information scarcity caused by the limited number of target samples; 3) domain discriminant adversarial loss to jointly align normal data features from the two domains. After removing all important parts, the basic feature extraction model "ResNet-50" with poor performance is obtained. First, aligning cross-domain standard samples through adversarial training can improve the performance of the basic model because the knowledge of the source domain is applicable to the target domain. Secondly, after adding the multi-spectral component fusion module M, more information can be integrated into the domain alignment, which improves the performance of most tasks. Thirdly, clustering normal data through center loss can achieve further improvement, which proves that compact feature representation is beneficial to distinguish normal instances from abnormal instances, while and “MsRA(w / oM)” are all inferior to MsRA, which illustrates the necessity of unifying these components.
[0170] Table 4
[0171]
[0172] Table 5 shows the effect of the number of spectral components on the model. Based on the performance of each spectral component, we select the spectral components with the Top-K highest performance to construct the multi-spectral features, where K∈{1, 2, 4, 8, 16, 32}. When K=1, it is equivalent to using global average pooling to process the features. It can be clearly seen that the performance of the multi-spectral features is higher than that of using only the lowest spectral component, which proves the effectiveness of the multi-spectral feature fusion module. The reason for the improved performance can be explained by the richer information introduced by multiple spectral components. In addition, according to the overall performance, it was found that K=8 had the best effect, and the Top-8 highest performing spectral components were used in all experiments.
[0173] Table 5
[0174]
[0175] In the method for detecting image abnormality of the present invention, Figure 1 Assume that only normal data in the source domain and a limited number of normal data in the target domain are accessible during training, and it is desired to implement anomaly data scoring function to detect abnormal targets. In the present invention, the anomaly score is defined as the distance from the instance to the center point of the normal instance. Figure 1 aDirectly applying a model trained on source domain normal data will result in many misclassifications of the target data since there may be a large domain drift between the two domains. Figure 1 bModels trained only on limited normal data in the target domain are prone to overfitting and thus perform poorly. Figure 1 c achieve better invention results by transferring rich knowledge from the source domain.
[0176] Figure 2 The multi-spectral cross-domain representation alignment method (MsRA) proposed in the present invention is shown. The feature extractor G is a deep convolutional neural network, which is used to generate a deep feature map for each input image. C is a classifier based on a multi-layer perceptron (MLP), which is trained using the cross entropy loss of normal data and fine-tunes the model. D is a domain discriminator, which is actually a second classifier used to distinguish between source domain data and target domain data. GRL is a gradient reversal layer. The present invention designs a multi-spectral feature fusion module M to integrate information from different spectral components, so as to make up for the problem of insufficient information caused by limited target training data, and can realize multi-spectral feature representation by weighted linear combination of K spectral components. In the multi-spectral feature space, through the adversarial loss function The domain discriminator D and feature extractor G and M are trained adversarially. In addition to In addition to learning domain-invariant features, the present invention also minimizes the loss Here, the orange and green points represent the features of the source domain data and the target domain data, respectively, and they are expressed in μ s and μ t for their respective centers.
[0177] Figure 3 The influence of each spectral component on the model performance is shown. In order to verify the design of fusing multiple spectral components, the effect of each individual spectral component is evaluated. Specifically, only one spectral component is used at a time to test the performance. Since the feature map size of the feature extractor is 7×7, the 2D DCT spectrum is divided into 7×7 parts. Figure 4 It can be clearly observed that: 1) lower spectral components produce better results than higher spectral components, and 2) some spectral components have poor performance. Here, it is not necessary to fuse all spectral components because most high-frequency components have a negative impact. One strategy is to select spectral components based on their performance. By fusing multiple spectral components, an AUROC of 85.5% is obtained on the task "Scissors", which is 4.95% higher than the best single spectral component.
[0178] Figure 4 The results of hyperparameter sensitivity analysis of MsRA on the Office-Home dataset are shown. The parameter sensitivity of MsRA is tested by changing the values of the parameters α∈{0.3, 0.5, 0.7, 1.0, 1.1, 1.3, 1.5} and β∈{0.1, 0.3, 0.5, 0.7, 0.9} used for balancing the loss. Figure 5 It is shown that MsRA is robust to fluctuations in both parameters and achieves competitive AUROC over a wide range of values.
[0179] The present invention is not limited by the above embodiments. The above embodiments and descriptions are only for explaining the principle of the present invention. Without departing from the concept and protection scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of the present invention to be protected is defined by the attached claims.
Claims
1. A method for detecting image anomalies by aligning multi-spectral cross-domain features in autonomous driving, characterized in that: The method for detecting image abnormalities comprises the following steps: Step 1, constructing a neural network based on the initialization feature extractor G, the multi-spectral feature fusion module M, the category classifier C and the domain discriminator D, including the following steps: Step 1.1, use the ResNet-50 network model pre-trained on ImageNet as the feature extractor G, and delete the global average pooling layer of ResNet-50; Step 1.2, use a fully connected layer to reduce the size of the feature extractor G to 256 dimensions; Step 1.3, design a multi-spectral feature fusion module M to integrate information from different spectral components, and connect the category classifier C, domain discriminator D, and center clustering module after the feature extractor G respectively; Step 2: Calculate the source domain center point μ s and the center point μ of the target domain t ; Step 3, obtaining the output of source domain samples and target domain samples, includes the following steps: Step 3.1: Randomly sample a small batch of source domain samples of size B With the target domain sample Step 3.2, and the source domain samples and target domain samples Input into the feature generator G to obtain the corresponding feature representation; Step 3.3, the generated feature representation is input into the category discriminator C, the domain discriminator D and the center clustering module respectively, and then the corresponding output is obtained, in which the number of target domain samples is much smaller than the number of source domain samples; Step 4: Calculate the category loss between the source domain sample model prediction and the target domain sample model prediction output and the true label Step 5: Calculate the domain adversarial loss between the predicted output and the true label of the source domain sample model and the target domain sample model for this batch Step 6: Calculate the distance loss between the predicted output of the source domain sample model and the target domain sample model and the samples with the true label for this batch As shown in formula (5): In the above formula (5), ||·||2 represents L2 regularization, μ s and μ t are the center points of the source domain and the target domain respectively; Step 7: Calculate the overall loss of the entire model As shown in formula (6), In the above formula (6), α and β are two parameters used to balance the loss; Step 8: Align source domain samples with target domain samples; Step 9, repeat steps 2 to 8 until the number of iterations reaches the maximum; Step 10, in the testing phase, calculate the feature center μ of all normal instances; Step 11: Calculate the test sample Anomaly score 2. The method for detecting image anomalies according to claim 1, characterized in that: The source domain center point μ calculated in step 2 s and the center point μ of the target domain t , including the following steps: Step 2.1: Based on the training data set, the source domain normal sample feature data after the multi-spectral feature fusion module M is input into the source domain normal sample feature center point calculation model to obtain the source domain normal sample feature center point μ s , the calculation model of the feature center point of the normal sample in the source domain is as follows (1): In the above formula (1): μ s is the feature center point of the source domain, n s is the number of normal samples in the source domain involved in training, represents the i-th training sample in the source domain, is the feature of the i-th instance of the current source domain calculated by the feature extractor G, It represents the output features of the normal samples in the source domain after the multi-spectral feature fusion module combines the information from different spectral components; Step 2.2: Input the target domain normal sample data into the target domain feature center point calculation model to obtain the target domain normal sample feature center point μ t , the calculation model of the center point of the normal sample in the target domain is as follows (2): In the above formula (2): μ t is the feature center point of the target domain, n t is the number of normal samples participating in training in the target domain, represents the jth training sample in the target domain, is the feature of the jth instance of the target domain calculated by the feature extractor G, It represents the output features of normal samples in the target domain after the multi-spectral feature fusion module combines information from different spectral components.
3. The method for detecting image anomalies according to claim 1, characterized in that: Step 4 calculates the category loss between the source domain sample model prediction and the target domain sample model prediction output and the true label In step 3, the source domain samples and target domain samples Under the supervision of the annotation information, the dual-domain joint cross entropy loss function is used to calculate the category loss between the source domain sample model prediction and the target domain sample model prediction output and the true label of this batch. As shown in the following formula (3): In the above formula (3): n s is the number of normal samples in the source domain participating in training, n t is the number of normal samples participating in training in the target domain, It represents the output features of the normal samples in the source domain that are combined with information from different spectral components after passing through the multi-spectral feature fusion module. It represents the output features of the target domain normal samples after the multi-spectral feature fusion module combines the information from different spectral components, and C is the category classifier.
4. The method for detecting image anomalies according to claim 1, characterized in that: Step 5, under the supervision of the labeling information of the source domain samples and the target domain samples in step 4, uses the multi-spectral domain adversarial loss function to calculate the domain adversarial loss between the predicted output of the source domain sample model and the target domain sample model of this batch and the true label. As shown in formula (4): In the above formula (4): It represents the expected value of the loss calculated by the discriminator on the output of the source domain data of the current batch input.
5. The method for detecting image anomalies according to claim 1, characterized in that: In step 10, the feature center μ of all normal instances is calculated as follows: In the above formula (7), n s is the number of instances in the source domain that participate in training, is the feature of the i-th instance of the source domain in the current batch calculated by the feature extractor G, It means that the output features from different spectral component information are integrated after passing through the multi-spectral feature fusion module.
6. The method for detecting image anomalies according to claim 1, characterized in that: Calculate the test sample as described in step 11 Anomaly score As shown in formula (8): In the above formula (8), is the feature of the mth instance of the test set in the current batch calculated by the feature extractor G, It represents the output features from different spectral components after the multi-spectral feature fusion module, and μ represents the feature center of the normal instance.
Citation Information
Patent Citations
Image anomaly detection method in combination with CNN migration learning and SVDD
CN107563431A
Retina abnormal detection method, device and equipment and storage medium
CN109833025A
Rolling bearing fault diagnosis method for improving model migration strategy
CN111721536A
Image classification method based on adversarial fusion multi-source transfer learning
CN111738315A
