Foreign object detection method and storage medium based on positive samples

Abstract

This invention discloses a foreign object detection method and storage medium based on positive samples. The method includes acquiring a test image and a template image of the same part of the device under test using a robot-based fixed-point inspection and imaging method; registering the test image and the template image; then using the foreign object detection region of the registered test image as input to a pre-trained foreign object detection network to obtain an accurate foreign object segmentation map; and finally, post-processing the foreign object segmentation map to obtain the foreign object region and probability. This invention uses a robot-based fixed-point imaging method to acquire samples, simplifying the difficulty of detection and localization, and is suitable for foreign object detection methods based on positive sample reconstruction and discrimination. Only normal samples need to be collected without manual annotation, greatly reducing the difficulty of collecting and creating training samples. Furthermore, the foreign object detection algorithm based on positive samples, by using a foreign object simulation generator, a foreign object reconstruction network, and a foreign object segmentation and discrimination network, is applicable to all types of foreign objects, with high detection accuracy and strong generalization ability.

Description

Technical Field

[0001] This invention relates to the field of visual surface anomaly detection technology, specifically to a foreign object detection method and storage medium based on positive samples. Background Technology

[0002] Currently, with the advancement and development of intelligent transportation, rail transit is being used in more and more cities. As a crucial component of rail transit, trains are susceptible to obstruction or clogging by debris such as litter, hanging objects, and leaves during operation, posing a potential hazard to their normal operation. Therefore, detecting and identifying components that may contain foreign objects is essential for identifying such objects, eliminating potential hazards, and is of great significance to the safety of rail transit.

[0003] Traditional maintenance methods rely on nighttime inspections by maintenance personnel. However, the electrified track system poses safety hazards, and relying solely on the experience of maintenance personnel is insufficient to promptly eliminate potential risks. Therefore, the development of automated foreign object (FOO) identification methods is urgently needed. Currently, FEO identification methods can be categorized historically into pre-deep learning and post-deep learning methods, among which:

[0004] Before deep learning, the main problem in traditional image processing research was obtaining shallow image features, such as grayscale values, SIFT, and HOG, and attempting to develop different detection methods based on statistical or traditional machine learning mechanisms, such as density estimation and single-class classification. Then, if the image or features did not conform to the corresponding distribution model, they would be identified as anomalies. However, the environment at railway sites is relatively complex, requiring numerous and specific image preprocessing steps with poor robustness, and the computational cost of various algorithms is staggering.

[0005] With the development of deep learning, methods for foreign object detection can be divided into supervised and unsupervised approaches. Supervised deep learning anomaly detection models, such as object detection, have high recognition rates for scenes with clearly defined anomaly types, and network training is relatively simple. However, their drawbacks include:

[0006] First, it requires a large number of samples for learning. Training requires at least several thousand or even tens of thousands of data points, which is costly. Moreover, due to the scarcity of foreign object samples in the scene, the workload of collecting samples is too large, and manual annotation may be too time-consuming.

[0007] Secondly, object detection has very high requirements for categories, and the model cannot exhaustively list all anomaly types, which has limitations.

[0008] Unsupervised autoencoders and GAN reconstruction methods can learn a powerful reconstruction subspace using anomaly-free images. However, learning a model solely from anomaly-free data lacks explicit optimization for discriminating anomaly detection, and determining the existence of anomalies that are not significantly different from normal appearances remains challenging, as some anomalies can often be reconstructed well. Furthermore, using synthetic anomalies to train discriminating segmentation methods can lead to overfitting to synthetic appearances, preventing the learned decision boundaries from generalizing well to real anomalies. Summary of the Invention

[0009] The present invention proposes a foreign object detection method based on positive samples, which can solve the above-mentioned technical problems.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A foreign object detection method based on positive samples includes the following steps:

[0012] S1. Use a robot to capture images at fixed points to obtain images of the device under test and template images of the same part of the device under test.

[0013] S2. Register the image to be tested with the template image to obtain the foreign object detection area in the image to be tested;

[0014] S3. Input the foreign object detection area into the pre-trained foreign object detection network to obtain an accurate foreign object segmentation map;

[0015] S4. Post-process the segmented foreign object image to perform anomaly localization and detection.

[0016] Furthermore, step S3 uses the foreign object detection area as input to a pre-trained foreign object detection network to obtain an accurate foreign object segmentation map, and also includes...

[0017] The foreign object detection area is enhanced in the original image by means of translation, rotation, scaling and changing the light intensity. Then the enhanced foreign object detection area is converted into a 1024*1024 size image and used as the input of the foreign object detection network.

[0018] Furthermore, the pre-trained foreign object detection network includes an anomaly simulation generator, a foreign object reconstruction network, and a foreign object segmentation and discrimination network. The processing steps of the anomaly simulation generator are as follows:

[0019] 6) Use the Berlin noise generator to generate noise and capture various shape anomalies;

[0020] 7) By randomly and uniformly selecting a threshold, the noise is binarized into an anomaly map;

[0021] 8) An anomalous texture source image A sampled from an anomalous source image dataset that is independent of the input image distribution;

[0022] 9) Using a random enhancement sampling method, three operations are randomly selected from the set {post-processing, sharpness, daylighting, equalization, brightness change, color change, automatic contrast} to enhance A;

[0023] 10) The enhanced texture image A, the anomaly image Ma, and the original image I are blended to generate an anomaly that just exceeds the distribution; the final simulated anomaly generator generates an anomaly image Ia defined as follows:

[0024]

[0025] Where A is the enhanced texture image, M a I represents the anomaly mask image, and I represents the original input image. It is M a The image is inverted, ⊙ is the element-wise multiplication operation, and β is the opacity parameter in the blending, which is uniformly sampled from the interval [0.1, 1.0].

[0026] Furthermore, the foreign object reconstruction network adopts an encoder-decoder architecture, wherein the foreign object detection network uses an encoder-decoder architecture, the encoder includes a first to a fourth convolutional layer and a pooling layer connected in sequence; the decoder includes a first to a fourth convolutional transform layer and an upsampling layer connected in sequence, and a fifth convolutional layer, the output of the fifth convolutional layer is used as the reconstructed image Ir output;

[0027] The encoder contains multiple layers of convolutional layers, activation layers, and pooling layers, while the decoder contains multiple layers of convolutional layers, activation layers, and upsampling layers. Both the decoder and encoder learn the encoding of normal samples and then generate only those that are close to the original input from the dimensionality-reduced encoding.

[0028] Furthermore, the foreign object segmentation and discrimination network adopts an encoder-decoder architecture similar to U-Net with skip connections. The encoder includes first to fourth convolutional layers and pooling layers connected in sequence; the decoder includes first to fourth convolutional transform layers and upsampling layers connected in sequence, as well as a fifth convolutional layer. The encoder's fourth pooling layer upsampling is connected to the decoder's first convolutional transform layer, the encoder's third pooling layer upsampling is connected to the decoder's second convolutional transform layer, the encoder's second pooling layer upsampling is connected to the decoder's third convolutional transform layer, the encoder's first pooling layer upsampling is connected to the decoder's fourth convolutional transform layer, and the decoder's fifth convolutional layer outputs a foreign object segmentation mask.

[0029] The encoder contains multiple layers of convolutional layers, activation layers, and pooling layers, while the decoder contains multiple layers of convolutional layers, activation layers, and upsampling layers. Encoding and decoding employ skip-connected encoding to comprehensively capture the multi-scale distribution of the normal data distribution in the high-dimensional image space.

[0030] Furthermore, S4 performs post-processing on the foreign object segmentation image to perform anomaly localization and detection, including:

[0031] The post-processing of the foreign object segmentation image yields the foreign object probability and region. This involves directly locating the anomaly using an anomaly detection mask, while simultaneously smoothing the pixel-level anomaly detection mask using a mean filter convolutional layer. Local anomaly response information is aggregated, and the image-level anomaly score is calculated by taking the maximum value η of the smoothed anomaly score map for anomaly detection. η is defined as:

[0032] η = max(M) o *f sf×sf )

[0033] Where f sf×sf is a mean filter of size sf×sf, and * is a convolution operator.

[0034] Furthermore, the training steps for the foreign object detection network are as follows:

[0035] In the undercarriage environment of the train, normal images of each component are collected by a robot using a fixed-point shooting method at the points where foreign object detection is required;

[0036] Among them, normal images of each component are collected for foreign object training and template images for testing, and the foreign object detection areas of each component in the collected images are extracted as inputs to the foreign object detection network.

[0037] Foreign object detection is performed based on positive samples, without the need to define labels. The foreign object reconstruction network uses each component to collect foreign object samples to be detected, a Berlin noise generator, and abnormal texture source images that are unrelated to the distribution of the foreign object detection area sample input image to train the foreign object reconstruction network. Then, the foreign object segmentation and discrimination network uses the output of the foreign object reconstruction network and the original foreign object sample channel to train the network, thus obtaining the pre-trained foreign object detection network.

[0038] Foreign object detection networks do not need to simulate the actual appearance of anomalies in the target domain, but rather generate appearances that just deviate from the normal distribution, and then learn an appropriate distance function to identify anomalies by deviating from the normal distribution;

[0039] The foreign object detection network is divided into a foreign object reconstruction network and a foreign object segmentation and discrimination network. The loss function of the foreign object reconstruction network is:

[0040] L rec (I,Ir )=λ1L SSIM (I,I r )+λ2L GMS (I,I r )+L2(I,I r )

[0041] Among them I, I r These are the original image and the image generated by the foreign object reconstruction network, respectively. λ1 and λ2 are the hyperparameters balancing the reconstruction loss, and λ1L... SSIM (I,I r L GMS (I,I r L2(I,I) and L2(I,I) r ) represent the SSIM, GMS, and L2 loss functions for the original image and the reconstructed image, respectively, where L SSIM (I,I r ) is defined as:

[0042]

[0043] Where h and w are the height and width of image I, respectively, and Np is the number of pixels in I; SSIM(I,I r SSIM(i,j) is the SSIM value centered at image coordinates (i,j); r () is a number between 0 and 1. The larger the number, the smaller the difference between the output image and the distortion-free image, that is, the better the image quality. It is defined as:

[0044]

[0045] Where μ I , δ represents the average values ​​of I and Ir, respectively. I , δ represents the annotation difference between I and Ir, respectively. IIr c represents the covariance of I and Ir; c1 and c2 are constants to avoid systematic errors caused by distinct zero values. GMSD starts from image gradient information to extract structural information. Different structures will have different gradient changes during reconstruction.

[0046] L GMSD (I,I r ) is defined as:

[0047]

[0048] Where GMSD(I,I) r (i,j) are the GMSD values ​​of I and Ir centered at image coordinates (i,j), h and w are the height and width of image I, respectively, and Np is the number of pixels in I;

[0049] The focus loss function is used to enhance the localization accuracy of difficult cases, and the formula is defined as follows:

[0050] L seg (M,M a )=-(1-M a M) γ log(M a M)

[0051] Where Ma and M are the ground truth and the anomaly segmentation mask, respectively, the overall loss function of the foreign object detection network is:

[0052] L(I,I r M a M) = L rec (I,I r )+L seg (M,M a ).

[0053] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0054] As can be seen from the above technical solution, the foreign object detection method based on positive samples of the present invention uses a robot fixed-point imaging method to acquire samples, which simplifies the difficulty of detection and positioning, and is applicable to foreign object detection methods based on positive sample reconstruction and discrimination. Only normal samples need to be collected and no manual annotation is required, greatly reducing the difficulty of collecting and preparing training samples. Furthermore, the foreign object detection algorithm based on positive samples, by using a foreign object simulation generator, a foreign object reconstruction network, and a foreign object segmentation and discrimination network, is applicable to all types of foreign objects, has high detection accuracy, and strong generalization ability. Attached Figure Description

[0055] Figure 1 This is a flowchart of a foreign object detection method based on positive samples;

[0056] Figure 2 This is a structural diagram of a foreign object detection network;

[0057] Figure 3 This is a flowchart of the foreign object simulation generator and foreign object synthesis process;

[0058] Figure 4 It is a foreign object simulation generator that generates various abnormal effect images from a single texture image;

[0059] Figure 5 These are the image to be tested and the template image. Figure 5 (a) is the image to be tested, and 5(b) is the template image;

[0060] Figure 6It is the registered image to be tested and the template image. Figure 6 (a) is the image to be tested after registration, and 6(b) is the template image after registration;

[0061] Figure 7 These are images showing the detection results of normal and foreign object images. Figure 7 (a) is the test image. Figure 7 (b) is the reconstructed graph. Figure 7 (c) is the segmentation mask image. Figure 7 (d) is the JET format mask image;

[0062] Figure 8 This is a flowchart of the foreign object segmentation mask detection process. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0064] like Figure 1 As shown, this embodiment discloses a foreign object detection method based on positive samples, including the following steps S1 to S4:

[0065] S1. Use a robot to capture images at fixed points to obtain images of the device under test and template images of the same part of the device under test.

[0066] It should be noted that in this embodiment, sampling points are set up in the environment under the train's carriage. At each sampling point, the robot takes two images at a fixed point. During the capture of both images, the same gimbal distance, angle, and zoom factor are set, resulting in images with relatively consistent resolution and field of view. Figure 5 As shown.

[0067] S2. Register the image to be tested with the template image to obtain the foreign object detection area in the image to be tested;

[0068] It should be noted that, due to potential positioning deviations between the navigation and gimbal during robot-based fixed-point shooting, the image to be tested and a template image are registered to compensate for this deviation. This further aligns the corresponding pixels in the two images. After successful registration, the corresponding areas of the foreign object detection region in both the test and template images are extracted and used as the foreign object inspection region in the test image. The matched test image and template image are shown below. Figure 6 As shown in the figure. In this embodiment, image registration is performed between the image to be tested and the template image to reduce the impact of excessive pixel offset.

[0069] S3. Input the foreign object detection area into the pre-trained foreign object detection network to obtain an accurate foreign object segmentation map;

[0070] S4. Post-process the foreign object segmentation image to obtain the foreign object probability and region;

[0071] This embodiment utilizes a robot-based fixed-point imaging method to ensure consistent image capture perspectives, simplifying the identification process. The foreign object detection region in the registered image is used as input to a pre-trained foreign object detection network. A precise foreign object segmentation map is obtained using a reconstruction network and a segmentation discriminant network. Post-processing of the segmentation map yields the foreign object region and its probability, making it applicable to any type of foreign object detection and exhibiting superior generalization ability. Compared to traditional object detection algorithms, this method only requires training on normal samples, eliminating the need for foreign object sample collection and labeling. Compared to methods based on traditional registration difference, this method significantly improves detection and localization accuracy and robustness in complex environments.

[0072] In step S3 above, where the foreign object detection area is used as input to a pre-trained foreign object detection network to obtain an accurate foreign object segmentation map, the method further includes:

[0073] The foreign object detection region is enhanced in the original image by means of translation, rotation, scaling, and changes in illumination intensity to compensate for the impact of image registration errors on foreign object detection and improve the robustness of the foreign object detection network. Then, the enhanced foreign object detection region is converted into a 1024*1024 image and used as the input to the foreign object detection network.

[0074] Step S3 above: Input the foreign object detection area as the pre-trained foreign object detection network to obtain an accurate foreign object segmentation map, as follows. Figure 2 The structure diagram of the foreign object detection network is shown, which includes an anomaly simulation generator, a foreign object reconstruction network, and a foreign object segmentation and discrimination network.

[0075] It should be noted that the foreign object detection network does not need to simulate the actual appearance of an anomaly in the target domain, but rather generates an appearance that just deviates from the normal distribution, and then learns an appropriate distance function to identify anomalies by deviating from the normal distribution. The anomaly simulation generator follows these steps.

[0076] 11) Use the Berlin noise generator to generate noise, which can capture various shape anomalies.

[0077] 12) By randomly and uniformly selecting a threshold, the noise is binarized into an anomaly map.

[0078] 13) An anomalous texture source image A sampled from an anomalous source image dataset that is independent of the input image distribution.

[0079] 14) Using a similar random enhancement sampling method, three operations are randomly selected from the set {post-processing, sharpness, sunlight, equalization, brightness change, color change, automatic contrast} to enhance A.

[0080] 15) The enhanced texture image A, the anomaly image Ma, and the original image I are blended to generate anomalies that just exceed the distribution, thus helping to tighten the decision boundaries in the trained network. Therefore, the final simulated anomaly generator generating the anomaly image Ia is defined as follows, with the synthesis flowchart as shown in the figure. Figure 3 :

[0081]

[0082] Where A is the enhanced texture image, M a I represents the anomaly mask image, and I represents the original input image. It is M a Image inversion, ⊙ represents element-wise multiplication. β is the opacity parameter in the blending, uniformly sampled from the interval [0.1, 1.0], such as... Figure 4 .

[0083] like Figure 2 As shown, the foreign object reconstruction network adopts an encoder-decoder architecture, wherein the foreign object detection network uses an encoder-decoder architecture. The encoder includes a first to a fourth convolutional layer and a pooling layer connected in sequence; the decoder includes a first to a fourth convolutional transform layer and an upsampling layer connected in sequence, as well as a fifth convolutional layer. The output of the fifth convolutional layer is used as the reconstructed image Ir output.

[0084] The encoder contains multiple layers of convolutional layers, activation layers, and pooling layers, while the decoder contains multiple layers of convolutional layers, activation layers, and upsampling layers. Both the decoder and encoder learn the encoding of normal samples and then generate only those closest to the original input from the dimensionality-reduced encoding.

[0085] It should be noted that this method uses artificially damaged abnormal images Ia generated by an abnormality simulation generator as input to the foreign object reconstruction network, and uses the real normal regions in Ia to reconstruct normal samples, while implicitly detecting abnormal regions.

[0086] like Figure 2As shown, the foreign object segmentation and discrimination network adopts an encoder-decoder architecture similar to U-Net with skip connections. The encoder includes first to fourth convolutional layers and pooling layers connected in sequence. The decoder includes first to fourth convolutional transform layers and upsampling layers connected in sequence, as well as a fifth convolutional layer. The encoder's fourth pooling layer upsampling is connected to the decoder's first convolutional transform layer, the encoder's third pooling layer upsampling is connected to the decoder's second convolutional transform layer, the encoder's second pooling layer upsampling is connected to the decoder's third convolutional transform layer, the encoder's first pooling layer upsampling is connected to the decoder's fourth convolutional transform layer, and the decoder's fifth convolutional layer outputs a foreign object segmentation mask.

[0087] The encoder contains multiple layers of convolutional layers, activation layers, and pooling layers, while the decoder contains multiple layers of convolutional layers, activation layers, and upsampling layers. Encoding and decoding employ skip-connected encoding to comprehensively capture the multi-scale distribution of the normal data distribution in the high-dimensional image space.

[0088] It should be noted that in this method, the channel connection between the output Ir of the foreign object reconstruction network and the original input image I is used as the input Ic of the discriminant sub-network. Since the foreign object reconstruction network has the property of restoring normal samples, the joint appearance difference between I and Ir is very large, which provides the necessary information for anomaly segmentation. Through discrimination, an accurate foreign object segmentation mask map can be output.

[0089] like Figure 7 As shown, the post-processing of the foreign object segmentation image yields the foreign object probability and region. An anomaly detection mask is used for direct anomaly localization, while a mean filter convolutional layer smooths the pixel-level anomaly detection mask, aggregating local anomaly response information. The image-level anomaly score is calculated by taking the maximum value η of the smoothed anomaly score map for anomaly detection. η is defined as:

[0090] η = max(M) o *f sf×sf )

[0091] Where f sf×sf is a mean filter of size sf×sf, and * is the convolution operator. The detection results for normal and foreign object images are as follows: Figure 7 As shown.

[0092] In this embodiment, the foreign object detection network needs to be trained before foreign object detection of the device. Specifically:

[0093] In the undercarriage environment of the train, normal images of each component are collected by a robot using a fixed-point shooting method at the points where foreign object detection is required;

[0094] In this process, normal images of each component are collected for use as template images during foreign object training and testing. The foreign object detection area for each component in the collected images is extracted and used as input to the foreign object detection network.

[0095] This paper uses positive samples for foreign object detection without the need to define labels. The foreign object reconstruction network is trained by collecting foreign object samples to be detected, a Burmester noise generator, and abnormal texture source images that are unrelated to the distribution of the foreign object detection area sample input image. Then, the foreign object segmentation and discrimination network is trained by connecting the output of the foreign object reconstruction network with the original foreign object sample channel to obtain the pre-trained foreign object detection network.

[0096] The foreign object detection network does not need to simulate the actual appearance of the abnormality in the target domain, but rather generates an appearance that just deviates from the normal distribution, and then learns an appropriate distance function to identify the abnormality by deviating from the normal distribution.

[0097] The foreign object detection network is divided into a foreign object reconstruction network and a foreign object segmentation and discrimination network. The loss function of the foreign object reconstruction network is:

[0098] L rec (I,I r )=λ1L SSIM (I,I r )+λ2L GMS (I,I r )+L2(I,I r )

[0099] Among them I, I r These are the original image and the image generated by the foreign object reconstruction network, respectively. λ1 and λ2 are the hyperparameters balancing the reconstruction loss, and λ1L... SSIM (I,I r L GMS (I,I r L2(I,I) and L2(I,I) r ) represent the SSIM, GMS, and L2 loss functions for the original image and the reconstructed image, respectively, where L SSIM (I,I r ) is defined as:

[0100]

[0101] Where h and w are the height and width of image I, respectively, and Np is the number of pixels in I. SSIM(I, I r SSIM(i,j) is the SSIM value centered at image coordinates (i,j). r () is a number between 0 and 1. The larger the number, the smaller the difference between the output image and the distortion-free image, that is, the better the image quality. It is defined as:

[0102]

[0103] Where μ I , δ represents the average values ​​of I and Ir, respectively. I , These represent the annotation differences between I and Ir, respectively. This represents the covariance of I and Ir. c1 and c2 are constants to avoid systematic errors caused by values ​​that are clearly zero. GMSD starts from image gradient information to extract structural information; different structures will have different gradient changes during reconstruction.

[0104] L GMSD (I,I r ) is defined as:

[0105]

[0106] Where GMSD(I,I) r (i,j) are the GMSD values ​​of I and Ir centered at image coordinates (i,j), and h, w, and Np have the same meaning as above.

[0107] The focus loss function is used to enhance the localization accuracy of difficult cases, and the formula is defined as follows:

[0108] L seg (M,M a )=-(1-M a M) γ log(M a M)

[0109] Where Ma and M are the ground truth and the anomaly segmentation mask, respectively. Therefore, the overall loss function of the foreign object detection network is:

[0110] L(I,I r M a M) = L rec (I,I r )+L seg (M,M a )

[0111] The foreign object detection method of this embodiment will be specifically described below through a specific example:

[0112] 1) The robot takes pictures of the inspection points to be inspected according to the planned task, and uses them as template pictures.

[0113] 2) The robot begins its inspection, proceeding according to the planned task to the inspection point. Images are taken using the same gimbal distance, angle, and zoom level as in 1) as the test images. The test images and template images have relatively consistent resolution and similar fields of view. However, due to deviations in robot navigation and gimbal positioning, there is a certain offset between the test images and template images. Figure 4 As shown.

[0114] 3) Perform image registration between the image to be tested and the template image to further align the corresponding pixels in the two images, and extract the foreign object detection area in the image to be tested that corresponds to the foreign object in the template image as the foreign object detection area.

[0115] 4) Shift and enhance the foreign object inspection area, and set the image size to 1024*1024.

[0116] 5) Using the foreign object to be inspected area, an abnormal image is generated by an abnormal simulation generator and used as input to the foreign object reconstruction network to train the foreign object reconstruction sub-network.

[0117] 6) Connect the foreign object reconstruction network with the original foreign object inspection area channel as the input and output of the foreign object segmentation and discrimination network, and train to obtain the foreign object segmentation and discrimination sub-network.

[0118] 7) Recognition of the image to be tested: Input the foreign object detection area into the foreign object detection network; obtain an accurate foreign object segmentation mask image.

[0119] 8) Smooth the foreign object segmentation mask image and aggregate local abnormal response information.

[0120] 9) The maximum value of the smoothed anomaly score map is taken as the confidence level for foreign object detection.

[0121] 10) The location of the foreign object is determined by taking the bounding rectangle of the smoothed anomaly fraction map.

[0122] The above embodiments disclose a foreign object detection system based on positive samples. Based on these embodiments, the present invention also provides a corresponding electronic device and storage medium. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus. A computer program is stored in the storage medium, and when the program is executed by the processor, the processor performs the steps based on the foreign object detection system based on positive samples.

[0123] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the methods described in the above embodiments.

[0124] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0125] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A positive sample based foreign matter detection method characterized by, The method comprises the following steps of: S1, acquiring a to-be-tested image and a template image of the same part of the to-be-tested equipment by using a robot fixed-point shooting mode; S2, registering the to-be-tested image and the template image to obtain a foreign matter to-be-tested region in the to-be-tested image; S3, inputting the foreign matter to-be-tested region into a pre-trained foreign matter detection network to obtain an accurate foreign matter segmentation map; The pre-trained foreign matter detection network comprises an abnormality simulation generator, a foreign matter reconstruction network and a foreign matter segmentation discriminant network, wherein the processing steps of the abnormality simulation generator are as follows: 1) generating noise by using a Berlin noise generator to capture various shape abnormalities; 2) binarizing the noise into an abnormality graph by randomly and uniformly selecting a threshold; 3) sampling an abnormality texture source image A from an abnormality source image data set irrelevant to the distribution of the input image; 4) using a random enhancement sampling method to randomly select three operations from a set {post-processing, sharpness, sun exposure, equalization, brightness change, color change, automatic contrast} to enhance A; 5) the enhanced texture image A, the abnormality graph Ma and the original image I are mixed to generate an abnormality just beyond the distribution; the final abnormality simulation generator generates an abnormality image Ia defined as, where A is the enhanced post-texture image, is the abnormal mask image, I is the input original image, is image negation, is element-wise multiplication, and β is the opacity parameter in blending, uniformly sampled from the interval [0.1, 1.0]; The foreign matter reconstruction network adopts an encoding-decoding architecture, wherein the foreign matter detection network uses an encoding-decoding architecture, the encoder comprises first to fourth convolutional layers and pooling layers connected in sequence; the decoder comprises first to fourth convolutional transformation layers, up-sampling layers and a fifth convolutional layer, and the output of the fifth convolutional layer is taken as a reconstructed image Ir output; The encoder comprises multiple convolutional layers, activation layers and pooling layers, the decoder comprises multiple convolutional layers, activation layers and up-sampling layers, the encoder and the decoder both learn the encoding of normal samples, and then only the original input close to the original input can be generated from the dimension-reduced encoding; The foreign matter segmentation discriminant network adopts an encoding-decoding architecture similar to the U-Net architecture and having a skip connection, wherein the encoder comprises first to fourth convolutional layers and pooling layers connected in sequence; the decoder comprises first to fourth convolutional transformation layers, up-sampling layers and a fifth convolutional layer, and the fourth pooling layer of the encoder is connected to the first convolutional transformation layer of the decoder through up-sampling, the third pooling layer of the encoder is connected to the second convolutional transformation layer of the decoder through up-sampling, the second pooling layer of the encoder is connected to the third convolutional transformation layer of the decoder through up-sampling, the first pooling layer of the encoder is connected to the fourth convolutional transformation layer of the decoder through up-sampling, and the fifth convolutional layer of the decoder outputs a foreign matter segmentation mask graph; The encoder comprises multiple convolutional layers, activation layers and pooling layers, the decoder comprises multiple convolutional layers, activation layers and up-sampling layers, the encoding and decoding adopt a skip connection encoding, and the multi-scale distribution of the normal data distribution in the high-dimensional image space is comprehensively captured; S4, post-processing the foreign matter segmentation image to perform abnormality positioning and detection.

2. The positive sample based foreign matter detection method according to claim 1, characterized by: In step S3, the foreign matter to-be-tested region is input into the pre-trained foreign matter detection network to obtain an accurate foreign matter segmentation map, and the method further comprises the following steps of: The foreign matter detection region is enhanced in the original image by translation, rotation, scaling, changing light intensity, and the like, and then the enhanced foreign matter detection region is converted into a 1024*1024 size image as an input of the foreign matter detection network.

3. The positive sample based foreign matter detection method according to claim 1, characterized by: The S4 performs post-processing on the foreign matter segmentation image, and performs abnormal positioning and detection, including: The foreign matter segmentation image is post-processed to obtain a foreign matter probability and region, wherein the abnormality detection mask is used for direct abnormality positioning, and the mean filter convolution layer is used for smoothing the pixel-level abnormality detection mask, aggregating local abnormality response information, calculating the image-level abnormality score by taking the maximum value η of the smoothed abnormality score map, performing abnormality detection, and η is defined as: wherein is a mean filter of size is a convolution operator.

4. The positive sample based foreign matter detection method according to claim 1, characterized by: The training steps of the foreign matter detection network are as follows: In the train car under the box body environment, the normal images of each component are collected by using a robot to take pictures at the point where foreign matter detection is needed. Among them, the normal images of each component are used for foreign matter training and template images when testing, and the foreign matter detection regions of each component of the collected images are extracted as inputs of the foreign matter detection network. Foreign matter detection is performed based on positive samples without defining labels; wherein the foreign matter reconstruction network uses the foreign matter detection sample of each component, the Berlin noise generator, and the abnormal texture source image irrelevant to the distribution of the foreign matter detection region sample input image to train the foreign matter reconstruction network; then the foreign matter segmentation discriminant network uses the output of the foreign matter reconstruction network to connect with the original foreign matter detection sample channel to train the network, and obtains the pre-trained foreign matter detection network; The foreign matter detection network does not need to simulate the real abnormal appearance in the target domain, but generates an appearance just beyond the normal distribution, and then learns a proper distance function to identify abnormalities by deviating from the normal; The foreign matter detection network is divided into a foreign matter reconstruction network and a foreign matter segmentation discriminant network, wherein the loss function of the foreign matter reconstruction network is: where I, I r are the original image and the foreign matter reconstruction network generated image respectively, λ1, λ2 are the reconstruction loss balance hyperparameters, , and are the SSIM, GMS, L2 loss functions of the original image and the reconstructed image respectively, where is defined as: where h and w are the height and width of the image I, respectively, and Np is the number of pixels in I; (i,j) is the SSIM value centered at image coordinates (i,j); is a number between 0 and 1, the larger the better, defined as: wherein respectively represent the average value of I, Ir, respectively represent the signed difference of I, Ir, respectively represent the covariance of I, Ir; c1, c2 are constants to avoid systematic errors caused by division by zero, GMSD extracts structural information from image gradient information, and different structures have different gradient changes when they are reconstructed, is defined as: wherein (i,j) is the GMSD value of I and Ir centered at image coordinate (i,j), h and w are the height and width of image I, respectively, and Np is the number of pixels in I; The focal loss function is used to enhance the positioning accuracy of difficult examples, and the formula is defined as: Wherein Ma and M are ground truth and abnormality segmentation mask, respectively, and therefore the overall loss function of the foreign matter detection network is: 。 5. A computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to make the processor execute the steps of the method according to any one of claims 1 to 4.

Images