Image-based foreign object and target anomaly detection method, system and terminal device

Abstract

The application discloses a kind of foreign matter and target anomaly detection method, system and terminal equipment based on image, comprising: for foreign matter detection, feature map atlas is extracted to the image to be detected, reference image, and the difference of two feature map atlas is used to detect the foreign matter area coordinate in the image to be detected, and segmentation is obtained to preliminary detection foreign matter subgraph;Foreign matter area coordinate is mapped to reference subgraph obtained on reference image, and with preliminary detection foreign matter subgraph, it is composed into true and false identification image pair;For target anomaly detection, target reference subgraph is segmented on reference image, and target area coordinate is mapped to the image to be detected to obtain target suspected anomaly subgraph, and with target suspected anomaly subgraph, it is composed into true and false identification image pair;True and false identification image pair is cascaded true and false identification, and foreign matter or target anomaly detection in the image to be detected is completed.The application is applied to image processing, pattern recognition field, and whether the foreign matter, target in the image to be detected can be detected rapidly and with high accuracy rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, target detection and identification, in particular to an image-based foreign matter and target anomaly detection method, system and terminal device. BACKGROUND

[0002] In recent years, the rapid development of deep learning has made major breakthroughs in various aspects of computer vision, especially in the field of target detection. As a key aspect of computer vision, target detection aims to detect targets in images and is a basic element for tasks such as target identification, tracking and target segmentation.

[0003] Foreign matter detection and target anomaly detection technology is commonly used for inspection tasks in fixed scenes such as pipe networks, and belongs to the field of target detection. The detection targets are usually divided into two categories: one is foreign matter, which is an extraneous object in the scene, and its distinctive feature is that the type and form of the object are unknown; the other is target anomaly, which is a state anomaly of the target object, such as surface cracks, damage, shedding, color, anti-counterfeiting patterns and other unpredictable defect types and forms in the scene.

[0004] For target detection technology in the case of unknown types and forms of foreign matter and target anomaly, the current deep learning models disclosed in the world all have some significant shortcomings. For example, the accuracy of the deep model algorithm is highly dependent on the quality and quantity of the training set, and the collection of foreign matter and target anomaly samples is particularly difficult. There are many types of things in the world, and their postures are also very rich. As long as they are not objects in the scene, they can be considered as foreign matter, so it is impossible to collect all foreign objects as training samples. The same is true for target anomaly, such as cracks, skewing, peeling, ablation or forgery of target objects, which changes the color of the target, changes the position, and other unpredictable defects. At the same time, the accuracy of the deep learning model also has the well-known long-tail effect of the model, i.e. the recognition rate of a small number of objects is very low. Although various improvements have been made to deep models based on long-tail data sets in academia, their detection accuracy is still far from the strict requirements of practical applications. SUMMARY

[0005] In view of the above shortcomings of the prior art, the present application provides an image-based foreign matter and target anomaly detection method, system and terminal device, which can quickly and accurately detect foreign matter and target anomaly in the image to be detected.

[0006] To achieve the above purpose, the present application provides an image-based foreign matter and target anomaly detection method, which comprises the following steps:

[0007] Step 1, acquiring a reference image and an image to be detected, and performing noise reduction processing on the reference image and the image to be detected;

[0008] Step 2, aligning the to-be-detected image with the reference image to obtain a mapping matrix of the to-be-detected image and the reference image;

[0009] Step 3, comprising obtaining a preliminary foreign matter subgraph and obtaining a target reference subgraph:

[0010] The obtaining process of the preliminary foreign matter subgraph is: respectively performing feature extraction on the to-be-detected image and the reference image to obtain a feature map atlas of the to-be-detected image and a feature map atlas of the reference image, and preliminarily detecting a foreign matter region coordinate in the to-be-detected image by using the difference of feature points in the two feature map atlases, and then segmenting the to-be-detected image based on the foreign matter region coordinate to obtain a preliminary foreign matter subgraph;

[0011] The obtaining process of the target reference subgraph is: obtaining a target region coordinate by performing target detection on the reference image or directly anchoring the target, and then segmenting the reference image based on the target region coordinate to obtain a target reference subgraph;

[0012] Step 4, comprising foreign matter detection and target anomaly detection:

[0013] The process of the foreign matter detection is: mapping the foreign matter region coordinate to the reference image based on the mapping matrix, segmenting the reference image based on the mapped foreign matter region coordinate to obtain a reference subgraph, and then forming a true-false identification image pair by combining the reference subgraph with the corresponding preliminary foreign matter subgraph, and performing brightness consistency processing;

[0014] The process of the target anomaly detection is: mapping the target region coordinate to the to-be-detected image based on the mapping matrix, segmenting the to-be-detected image based on the mapped target region coordinate to obtain a target suspected anomaly subgraph, and then forming a true-false identification image pair by combining the target suspected anomaly subgraph with the corresponding target reference subgraph, and performing brightness consistency processing;

[0015] Step 5, performing cascade true-false identification on the true-false identification image pair to complete the foreign matter or target anomaly detection in the to-be-detected image.

[0016] To achieve the above purpose, the application also provides a foreign matter and target anomaly detection system based on images, which adopts the above method, and the detection system comprises:

[0017] An image preprocessing unit is configured to obtain a reference image and a to-be-detected image, and perform noise reduction processing on the reference image and the to-be-detected image;

[0018] An image alignment unit is configured to align the to-be-detected image with the reference image to obtain a mapping matrix of the to-be-detected image and the reference image.

[0019] A foreign matter segmentation unit is configured to respectively perform feature extraction on the to-be-detected image and the reference image to obtain a feature map atlas of the to-be-detected image and a feature map atlas of the reference image, and to preliminarily detect a foreign matter region coordinate in the to-be-detected image by using a difference between feature points in the two feature map atlases, and to segment the to-be-detected image based on the foreign matter region coordinate to obtain a preliminarily detected foreign matter sub-image.

[0020] A target anomaly segmentation unit is configured to perform target detection on the reference image or directly anchor a target to obtain a target region coordinate, and to segment the reference image based on the target region coordinate to obtain a target reference sub-image.

[0021] A brightness processing unit is configured to map the foreign matter region coordinate to the reference image according to the mapping matrix, or to map the target region coordinate to the to-be-detected image according to the mapping matrix, and to segment the reference image based on the mapped coordinate to obtain a reference sub-image or to segment the to-be-detected image based on the mapped coordinate to obtain a target suspected anomaly sub-image, and to form a true-false identification image pair by combining the reference sub-image with the preliminarily detected foreign matter sub-image or by combining the target suspected anomaly sub-image with the target reference sub-image, and to perform brightness consistency processing.

[0022] A cascaded true-false identification unit is configured to perform cascaded true-false identification on the true-false identification image pair to complete foreign matter or target anomaly detection in the to-be-detected image.

[0023] To achieve the above object, the application further provides a terminal device, wherein the terminal device is provided with:

[0024] A memory is configured to store a program.

[0025] A processor is configured to execute the program stored in the memory, and when the program is executed, the processor is configured to execute the method as described above.

[0026] Compared with the prior art, the application has the following beneficial technical effects:

[0027] 1.The application can not only quickly and accurately detect foreign objects and target abnormalities in the image, but also deeply overcome the problem that the accuracy of neural network model algorithm in foreign object and target abnormality detection depends on complete and balanced conditions such as complete types of foreign object samples, target defect abnormality samples, and rich postures.

[0028] 2.The application has been integrated into a robot, a drone, and other inspection equipment equipped with a camera, and can also be deployed on a product defect detection device on a production line, and has wide application in reality. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the drawings shown.

[0030] Figure 1 The flowchart of the image-based foreign object and target abnormality detection method in embodiment 1 of the present application;

[0031] Figure 2 The structural block diagram of the image-based foreign object and target abnormality detection system in embodiment 2 of the present application;

[0032] Figure 3 The structural block diagram of the terminal device in embodiment 3 of the present application.

[0033] The implementation, functional features and advantages of the present application will be further described with reference to the drawings. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0035] In addition, the technical solutions of each embodiment of the present application can be combined with each other, but it must be based on the fact that a person skilled in the art can realize it. When the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0036] Embodiment 1

[0037] The embodiment discloses an image-based foreign matter and target anomaly detection method, which can not only quickly and accurately detect foreign matters and target anomalies in a to-be-detected image, but also deeply overcome the defect that the accuracy of a neural network model algorithm in foreign matter and target anomaly detection seriously depends on complete foreign matter sample types, rich postures and balanced quantities, and the adaptability and high robustness of the method are completely not limited to unknown characteristics such as types, postures and quantities of foreign matters and target anomalies. The detection method in the embodiment can be applied to scenes such as household equipment maintenance and industrial product defect detection, and can be integrated on inspection equipment such as robots and unmanned aerial vehicles equipped with cameras, or deployed on product defect detection equipment on production lines. For example, the inspection and detection of important targets of underground pipeline networks in the household equipment field, the safety inspection and detection of power lines and power grids, and the defect detection of products in the industrial field.

[0038] Reference Figure 1 The image-based foreign matter and target anomaly detection method in the embodiment specifically includes the following steps:

[0039] Step 1, obtaining a reference image and a to-be-detected image, and performing noise reduction processing on the reference image and the to-be-detected image;

[0040] Step 2, aligning the to-be-detected image with the reference image to obtain a mapping matrix of the to-be-detected image and the reference image;

[0041] Step 3, including obtaining a preliminary detection foreign matter sub-image and obtaining a target reference sub-image:

[0042] The process of obtaining the preliminary detection foreign matter sub-image is as follows: feature extraction is performed on the to-be-detected image and the reference image respectively to obtain a feature map of the to-be-detected image and a feature map of the reference image, and the difference between feature points in the two feature maps is used to preliminarily detect the foreign matter region coordinates in the to-be-detected image, and then the preliminary detection foreign matter sub-image is segmented on the to-be-detected image based on the foreign matter region coordinates;

[0043] The process of obtaining the target reference sub-image is as follows: target region coordinates are obtained by target detection on the reference image or direct target anchoring, and then the target reference sub-image is segmented on the reference image based on the target region coordinates;

[0044] Step 4, including foreign matter detection and target anomaly detection:

[0045] The process of foreign matter detection is as follows: the foreign matter region coordinates are mapped to the reference image based on the mapping matrix, and the reference sub-image is segmented on the reference image based on the mapped foreign matter region coordinates, and then the reference sub-image and the corresponding preliminary detection foreign matter sub-image are combined to form a true-false identification image pair, and brightness consistency processing is performed.

[0046] The process of target anomaly detection is: mapping the target region coordinates to the image to be detected based on the mapping matrix, and segmenting the target suspected anomaly sub-image on the image to be detected based on the mapped target region coordinates, then forming a true-false identification image pair by combining the target suspected anomaly sub-image with the corresponding target reference sub-image, and performing brightness consistency processing;

[0047] Step 5: performing cascade true-false identification on the true-false identification image pair to complete the foreign matter or target anomaly detection in the image to be detected.

[0048] In this embodiment, the reference image refers to an image captured by a camera at a certain time node at a certain monitoring point. In this image scene, there is generally no foreign matter except the target of interest, and the target has no abnormal phenomenon. The reference image can be replaced by any image captured at the monitoring point. The image to be detected refers to an image captured by the same camera at the same monitoring point. It is necessary to detect and judge whether foreign matter or target anomaly occurs in the scene, so as to subsequently transmit the detection result back to the visualization interface, so that the staff or robot can maintain and repair at the monitoring point.

[0049] In the specific implementation process of step 1, the reference image and the image to be detected can be denoised by using a 2D Gaussian filter. It should be noted that in the specific application process, the reference image and the image to be detected can also be denoised by using mean filtering, morphological denoising, etc.

[0050] In the specific application process, although the reference image and the image to be detected are captured by the same camera, because the reference image and the image to be detected are obtained at different times, when capturing the two images, the absolute position of the camera may not be exactly the same, which causes the position values of the targets in the two image scenes to be different. Therefore, it is necessary to align the two images before foreign matter detection. In the specific implementation process, the image to be detected is aligned with the reference image, and the mapping matrix between the image to be detected and the reference image is obtained, which specifically includes the following steps:

[0051] Step 201: extracting key points in the reference image and the image to be detected respectively, and obtaining feature vectors of the key points;

[0052] Step 202: performing feature matching based on the feature vectors of the key points in the reference image and the image to be detected to obtain a plurality of key point pairs;

[0053] Step 203: performing cyclic iteration deletion on outliers in the key point pairs until a preset iteration termination condition is met;

[0054] In step 204, coordinate mapping is performed based on the remaining key point pairs to obtain a mapping matrix between the to-be-detected image and the reference image.

[0055] In a specific implementation, to make the feature vector of the key point have scale invariance, the embodiment extracts the key point by constructing a multi-layer pyramid image. That is, the original reference image and the to-be-detected image are scaled according to different scales, for example, the original image size is scaled to 1 / 1.2, 1 / 1.5, etc. of the original size. Then, it is determined whether a pixel neighborhood on a certain layer of the pyramid image is a predetermined key point. If not, it is determined whether the next pixel is a key point. Otherwise, the position, correlation value, neighborhood size, and image scale of the key point are recorded. Then, the position and neighborhood range of the key point are transformed to the image of other scales, and it is determined whether the point is a key point on the current scale image. If it is also determined to be a key point, the position and neighborhood range of the key point are transformed to the image of other scales for judgment. Otherwise, in the constructed multi-layer pyramid image, the current key point is transformed by scaling. As long as it is determined to be not a key point on any scale (layer) image, the point is not a key point, and the next pixel is replaced to continue the judgment of the key point. If the current pyramid image determines that the point is a key point, the extreme value is taken from the correlation values recorded at each scale, and the position and scale of the key point are recorded. In this way, the image is traversed until all the key points and their positions and scales are extracted. Finally, according to the scale information recorded in each key point, the scale with the most occurrences is found. The key point on this scale is the final key point to be extracted, and the scale image is the best image for extracting the feature vector.

[0056] In a specific application, the way of extracting the key point in a certain scale image of the multi-layer pyramid image is as follows:

[0057] ;

[0058] wherein, 、 is the coordinate position of the image pixel point, is the neighborhood size of the point , which is generally an odd number; is the weight of the pixel point from the pixel point , which is related to the relative distance between the two pixel points; is the pixel value of the selected point , is the pixel value of the point , is the absolute difference between the pixel values of the point and the point , The number of key points in the image, whether it is a key point threshold.

[0059] After determining the optimal scale image for extracting the feature vector, the process of extracting the key points on the scale image is as follows:

[0060] First, a neighborhood centered on a certain key point is extracted on the scale image, and a pair of pixel points is randomly obtained in the neighborhood. The pixel value of the pixel point pair is obtained.

[0061] Then, the sum of the pixel values in the neighborhood of the points in any pixel point pair is calculated as the pixel value of the point, thereby reducing noise interference. Then, a feature descriptor is obtained using the sum of the pixel values in the neighborhood of the two points in the pixel point pair.

[0062]

[0063]

[0064] wherein, is the neighborhood pixel value of the pixel point is the neighborhood pixel value of the pixel point , i.e.

[0065]

[0066] wherein, is the pixel value of the point

[0067] Finally, the feature descriptors of the pixel point pairs are concatenated to obtain the feature vector of the key point, which is:

[0068]

[0069] wherein, is the feature vector of the key point; is the number of pixel point pairs, is the dimension feature in the pixel point feature vector.

[0070] As a preferred embodiment, in order to make the feature vector of the key point rotationally invariant, the following operations are performed after selecting the pixel point pair:

[0071] First, the pixel point pair is rotated by an angle of 90 degrees. ​​​​​​​​​​​​After the pixel pair, the pixel coordinates of the pixel pair set are converted into the following matrix form:

[0072]

[0073] wherein, is the pixel coordinate matrix of the pixel pair set;

[0074] Then, the principal direction of the key point is used to obtain the rotation matrix, which is:

[0075]

[0076] wherein, is the rotation matrix, is the principal direction of the key point;

[0077] Then, the pixel coordinate matrix of the pixel pair set is rotated by using the rotation matrix to obtain the rotated pixel pair set matrix, which is:

[0078]

[0079] wherein, is the rotated pixel pair set matrix, is the matrix multiplication;

[0080] Finally, the feature vector of the pixel is calculated based on the pixel pair in the pixel pair set matrix , that is:

[0081] .

[0082] At this point, the feature vector of the key point has rotation invariance.

[0083] In the specific implementation process, when the gray value of a key point deviates from the centroid of the neighborhood, the principal direction of the key point can be obtained through the vector from the key point to the centroid, and the calculation process is as follows:

[0084] First, the geometric moment of the image is calculated, which is wherein, is a non-negative number, the sum of which is the order of the geometric moment, such as is the second moment of the image;

[0085] Then, the centroid of the neighborhood image is obtained by using the 0th moment and the 1st moment, which is .

[0086] Finally, the principal direction of the key point can be calculated.

[0087] In the implementation process, the feature matching process of the feature vector of the key point is specifically:

[0088] First, the Hamming distance between two key point feature vectors is calculated. It is:

[0089] ;

[0090] Wherein, , are the feature vectors of a key point pair in the reference image and the image to be detected respectively, is the Hamming distance between the two feature vectors, is also or calculated;

[0091] The smaller the value of , the higher the similarity of the two feature vectors, and vice versa. Therefore, when the value of satisfies the predetermined threshold , it is determined that the key points in the reference image and the image to be detected are matched successfully, so as to obtain the coordinate values of the corresponding key points in the two images , which are:

[0092]

[0093] Wherein, are the coordinates of the matched key points in the reference image and the image to be detected respectively.

[0094] In order to ensure the accuracy of the subsequent image alignment, the matched key point pairs need to be further screened and deleted. In this embodiment, the second derivatives of the matched key points in the x direction and the y direction, as well as their variance and mean value, should be within a reasonable range as the reference, and the key point pairs are screened through loop iteration. The specific process is:

[0095] First, the first derivative of the key point pair in the x and y directions is calculated, which is:

[0096] ;

[0097] Wherein, , are the values of the two key points in the x direction of the key point pair, is the first derivative of the key point pair in the x direction, , are the values of the two key points in the y direction of the key point pair, is the first derivative of the key point pair in the y direction, is the number of key point pairs;

[0098] Then, the mean and standard deviation of the first order derivative of all key point pairs in the x and y directions are calculated, and are:

[0099]

[0100]

[0101] wherein, is the mean of the first order derivative of all key point pairs in the x direction, is the standard deviation of the first order derivative of all key point pairs in the x direction, is the mean of the first order derivative of all key point pairs in the y direction, is the standard deviation of the first order derivative of all key point pairs in the y direction;

[0102] Finally, the discrete points are screened and iteratively deleted in the x and y directions, respectively.

[0103] In this embodiment, the process of screening and iteratively deleting the discrete points in the x direction is as follows:

[0104] Step X1, the first order derivative sequence of the key point pairs in the x direction is calculated The discrete degree sequence of the mean of the sequence is

[0105]

[0106] Step X2, the mean and the standard deviation of the discrete degree sequence of all key point pairs in the x direction are calculated, and are:

[0107]

[0108] Step X3, the lower threshold and the upper threshold of the discrete degree sequence of the key point pairs in the x direction are calculated according to the mean and the standard deviation , and are:

[0109]

[0110]

[0111] Step X4, the key point pairs satisfying are retained, and the other key point pairs are deleted, and then the mean and the standard deviation of the discrete degree sequence of all remaining key point pairs in the x direction are calculated, and it is determined whether ​​​​​​Is the difference less than the set threshold?

[0112] If so, then end the iterative deletion of discrete points in the x-direction and output the remaining keypoint pairs;

[0113] Otherwise, let 、 Then, return to step X3.

[0114] In this embodiment, the process of filtering discrete points in the y-direction and iteratively deleting them is as follows:

[0115] Step Y1: Obtain all keypoint pairs after iterative deletion of discrete points in the x-direction, and then calculate the first derivative sequence of the keypoint pairs in the y-direction. relative to the mean of the sequence discrete sequence ,for:

[0116]

[0117] Step Y2: Calculate the mean of the dispersion sequence of all keypoint pairs in the y-direction. and standard deviation ,for:

[0118] ;

[0119] in, This represents the number of keypoint pairs remaining after iterative deletion of discrete points along the x-direction.

[0120] Step Y3, based on the mean and standard deviation Calculate the lower limit threshold of the dispersion sequence of keypoint pairs in the y-direction. With upper limit threshold ,for:

[0121] ;

[0122] .

[0123] Step Y4, retain those that satisfy After removing other keypoint pairs, calculate the mean of the dispersion sequence of the remaining keypoint pairs in the y-direction. and standard deviation and judge and Is the difference less than the set threshold?

[0124] If so, then end the iterative deletion of discrete points in the y-direction and output the remaining keypoint pairs;

[0125] Otherwise, let 、 Then, return to step Y3.

[0126] It is worth noting that in the specific application process, the iterative deletion of discrete points in the y direction can also be performed first, and then the iterative deletion of discrete points in the x direction.

[0127] After completing the iterative deletion of discrete points in the x direction and the y direction, the coordinate mapping can be performed according to the remaining key point pairs, and the mapping matrix between the to-be-detected image and the reference image is obtained through the key point coordinates in the to-be-detected image and the coordinates of the corresponding key points in the reference image, which is:

[0128]

[0129] wherein x and y are the key point coordinates on the reference image, is a scaling factor, is a mapping matrix, is a scaling factor, is a scaling factor, , is a key point coordinate on the to-be-detected image;

[0130] Using the mapping matrix, the pixel coordinates on the to-be-detected image can be mapped to the corresponding positions on the reference image, and thus the alignment of the reference image and the to-be-detected image is completed.

[0131] In the specific implementation process of step 3, the process of extracting the feature map of the to-be-detected image and the feature map of the reference image is specifically:

[0132] Step 301, the coordinates of the four vertices of the common target region in the two images are extracted, and then the vertex coordinates are used to cut the reference image and the to-be-detected image respectively to obtain the aligned reference image and to-be-detected image;

[0133] Step 302, after the reference image and the to-be-detected image are aligned, the gabor features of the reference image and the gabor features of the to-be-detected image are extracted respectively, which are taken as the feature maps of the reference image and the to-be-detected image respectively;

[0134] Step 303, the sift features of the feature maps of the reference image and the to-be-detected image are extracted respectively to obtain the sift feature maps of the reference image and the to-be-detected image, and the sift feature maps of the reference image and the to-be-detected image are normalized;

[0135] Step 304, the feature similarity of each feature point in the two normalized sift feature maps is calculated to obtain the similarity of each feature point in the two sift feature maps ​​

[0136] ;

[0137] wherein, 、 are the sift feature maps of the extracted reference image and the aligned to-be-detected image respectively; the range of , if the value is closer to the value 0, the feature vector similarity is poor, and it can be determined that the point is a point on the foreign matter; if it is closer to the value 1, the similarity of the corresponding points in the two images is high, and it is a normal point;

[0138] Step 305, a pure black image completely same as the size of the to-be-detected image is constructed, the points with feature similarity less than the set threshold value in the sift feature map of the to-be-detected image are screened out, the coordinates of the screened points are determined, and the pixel value of the corresponding coordinate position in the pure black image is set to 225 (i.e. white), thereby obtaining a black and white binary image, wherein the set threshold value of the feature similarity is an empirical value, and can be set to 0.8 to ensure that no missed detection phenomenon occurs;

[0139] Step 306, the obtained black and white binary image is subjected to erosion and expansion operation, and then connected domain algorithm or flood fill algorithm is used for series connection, so as to connect the white pixels into one or more regions, thereby preliminarily detecting the foreign matter region coordinates in the to-be-detected image, and then based on the foreign matter region coordinates, the to-be-detected image can be segmented to obtain the preliminarily detected foreign matter sub-image.

[0140] Since the reference image and the to-be-detected image are photographed in different time periods, if the ambient light is inconsistent or the light compensation is different, the illumination in the two images will be inconsistent, which will lead to the false alarm rate of the foreign matter or target abnormality. Therefore, it is necessary to perform image brightness consistency processing on the segmented foreign matter sub-image or target suspected abnormal sub-image before authenticity identification. In this embodiment, the process of brightness consistency processing is specifically as follows:

[0141] The authenticity identification image pair is converted into a gray image pair, and the gray mean values 、 and normalized brightness histograms of the two images are calculated respectively, that is:

[0142] ;

[0143] wherein, 、 are the histograms of the preliminarily detected foreign matter sub-image or target suspected abnormal sub-image and the reference sub-image or target reference sub-image in the authenticity identification image pair respectively;

[0144] According to the normalized brightness histogram, the difference distance is calculated, that is:​

[0145] ;

[0146] The value range is [0, 1], when the value is 0, it means that the two images are completely consistent, and when the value exceeds a certain value, it means that there is an obvious brightness difference between the two images in the true and false identification image pair, and the brightness consistency processing needs to be performed.

[0147] Then, according to the relationship between the threshold value and the threshold value , , the brightness consistency processing is performed on the preliminary inspection foreign matter sub-image or the target suspected abnormal sub-image in the true and false identification image pair.

[0148] When , the process of performing the brightness consistency processing on the preliminary inspection foreign matter sub-image or the target suspected abnormal sub-image in the true and false identification image pair is as follows:

[0149] First, according to the histograms of the preliminary inspection foreign matter sub-image or the target suspected abnormal sub-image and the reference sub-image in the true and false identification image pair, the normalized histograms of the preliminary inspection foreign matter sub-image or the target suspected abnormal sub-image and the reference sub-image are calculated , , which are as follows:

[0150] ;

[0151] Among them, , are the pixel numbers of the preliminary inspection foreign matter sub-image or the target suspected abnormal sub-image and the reference sub-image or the target reference sub-image, respectively;

[0152] Then, the cumulative distribution functions of the normalized histograms are calculated , , which are as follows:

[0153] .

[0154] Then, based on the cumulative distribution functions, the mapping function of the brightness consistency of the preliminary inspection foreign matter sub-image or the target suspected abnormal sub-image to the reference sub-image or the target reference sub-image in the true and false identification image pair is obtained , which is as follows:

[0155]

[0156] Finally, according to the mapping function , the brightness consistency processing is performed on the preliminary inspection foreign matter sub-image or the target suspected abnormal sub-image, that is:

[0157] ;

[0158] Among them, This is a preliminary sub-image of foreign objects or suspected abnormal targets before brightness consistency processing. This is a preliminary sub-image of foreign objects or suspected anomalies after brightness consistency processing. These are the coordinates of the pixels.

[0159] when The process of brightness consistency processing for the initial foreign object sub-image or suspected abnormal target sub-image in the image pair for authenticity identification is as follows:

[0160] First, based on the grayscale image mean of the genuine / fake identification image pair... , Calculate the brightness consistency processing coefficient , ,for:

[0161] ;

[0162] .

[0163] Then, based on the brightness consistency processing coefficient , Brightness consistency processing is performed on the initial foreign object sub-image or the sub-image of the suspected target anomaly, that is:

[0164]

[0165] Because the threshold for feature similarity in step 305 is set too low and the illumination difference between the genuine and fake image pairs is large, there will be a certain false alarm rate in the initial detection of foreign objects or target anomalies. Therefore, it is necessary to perform authenticity verification on the initially detected foreign objects or suspected target anomaly sub-images. In order to ensure a high detection accuracy rate, this embodiment adopts a cascaded authenticity verification method in step 5 to verify the authenticity of the initially detected foreign objects or target anomalies. The specific implementation process includes two parts: authenticity verification based on multiple feature fusion and authenticity verification based on deep neural networks.

[0166] In practical implementation, authenticity verification based on the fusion of multiple features specifically includes:

[0167] The true and false identification images that have undergone brightness consistency processing are size-uniformed, that is, the reference sub-image or target reference sub-image is scaled to the same size as the initial foreign object sub-image or the target suspected abnormal sub-image through bilinear interpolation.

[0168] Based on grayscale images, extract the illumination difference features between the centering reference sub-image and the initial foreign object sub-image for authenticity identification, or between the target reference sub-image and the target suspected abnormality sub-image. Contrast difference characteristics Target contour difference features , wherein

[0169]

[0170] , wherein , is the mean and variance of the gray scale of the initial foreign matter subgraph or the target suspected abnormal subgraph after the brightness consistency processing, , is the mean and variance of the gray scale of the reference subgraph or the target reference subgraph, is the covariance of the true and false identification image pair, is the number of pixel points of the initial foreign matter subgraph or the target suspected abnormal subgraph, the reference subgraph or the target reference subgraph in the x direction, is the number of pixel points of the initial foreign matter subgraph or the target suspected abnormal subgraph, the reference subgraph or the target reference subgraph in the y direction, is the reference subgraph or the target reference subgraph, is the initial foreign matter subgraph or the target suspected abnormal subgraph after the brightness consistency processing, , is the weight coefficient;

[0171] Based on the illumination brightness difference feature vector , the contrast difference feature vector , and the target contour difference feature vector , the similarity distance measure is calculated is:

[0172]

[0173] wherein “ ” is the vector dot product, the value ranges between [0, 1], when the value is 1, the two images in the true and false identification image pair are completely the same, when the value is 0, the two images in the true and false identification image pair are completely different images;

[0174] determines whether it is lower than the first foreign matter threshold or the first target abnormal threshold: if yes, it is determined that the foreign matter in the current initial foreign matter subgraph is a real foreign matter or a target abnormality; otherwise, it is determined that the foreign matter in the current initial foreign matter subgraph is a false foreign matter or the current initial target abnormal subgraph is a target suspected abnormality.

[0175] After the true and false identification based on the fusion of multiple features, the true and false identification image pair identified as a false foreign matter or a target suspected abnormality will be subjected to a secondary judgment, i.e., true and false identification based on a deep neural network, specifically:

[0176] The authenticity identification based on the deep neural network uses a neural network to train a feature classification extractor. Currently, classic feature classifiers such as alexnet, resnet, vision transformer, etc. network model can be directly used to train the feature extractor, or a feature extraction and classification model can be designed independently.

[0177] The trained feature classification extractor is used to extract features from the authenticity identification image pair determined as a false foreign object or the authenticity identification image pair of the target suspected abnormality. Then, the extracted features are evaluated for similarity. In this embodiment, the cosine distance is used as the similarity evaluation metric. The closer the metric value is to 0, the higher the credibility of the false foreign object being a real foreign object or the target being normal. Otherwise, the higher the credibility of the false foreign object being a real foreign object or the target being abnormal. For example, when the similarity evaluation metric is less than the second foreign object threshold, it is determined that the false foreign object is a real foreign object. When the similarity evaluation metric is less than the second target abnormality threshold, it is determined that the target state is abnormal.

[0178] At this point, the foreign object or target abnormality detection is completed, and the high-credibility foreign object or target abnormality state can be output.

[0179] It is worth noting that although each step in the embodiment Figure 1 is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in the embodiment may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0180] Embodiment 2

[0181] Based on the image-based foreign object and target abnormality detection method in embodiment 1, this embodiment discloses an image-based foreign object and target abnormality detection system. Referring to Figure 2 , the image-based foreign object and target abnormality detection system includes an image preprocessing unit, an image alignment unit, a foreign object segmentation unit, a target abnormality segmentation unit, a brightness processing unit, and a cascaded authenticity identification unit. Specifically:

[0182] The image preprocessing unit is used to acquire reference images and to-be-detected images, and to perform noise reduction processing on the reference images and to-be-detected images.

[0183] The image alignment unit is used to align the image to be detected with the reference image to obtain a mapping matrix between the image to be detected and the reference image;

[0184] The foreign object segmentation unit is used to extract features from the image to be detected and the reference image respectively, to obtain the feature map of the image to be detected and the feature map of the reference image. The difference between the feature points in the two feature maps is used to initially detect the coordinates of the foreign object region in the image to be detected. Then, based on the coordinates of the foreign object region, the image to be detected is segmented to obtain the initial foreign object sub-image.

[0185] The target anomaly segmentation unit is used to perform target detection on the reference image or directly anchor the target to obtain the target region coordinates, and then segment the target reference image based on the target region coordinates to obtain the target reference sub-image;

[0186] The brightness processing unit is used to map the coordinates of the foreign object region onto the reference image according to the mapping matrix, or to map the coordinates of the target region onto the image to be detected according to the mapping matrix, and to segment the reference sub-image on the reference image or to segment the target suspected abnormal sub-image on the image to be detected based on the mapped coordinates. Then, the reference sub-image and the corresponding initial detection foreign object sub-image, or the target suspected abnormal sub-image and the corresponding target reference sub-image are combined to form a true / false identification image pair, and brightness consistency processing is performed.

[0187] The cascaded authenticity identification unit is used to perform cascaded authenticity identification on pairs of authentic and fake images, and complete the detection of foreign objects or target anomalies in the image to be detected.

[0188] In this embodiment, the specific working process and working principle of the image preprocessing unit, image alignment unit, foreign object segmentation unit, brightness processing unit, and cascaded authenticity identification unit are the same as those in Embodiment 1, and therefore will not be described again in this embodiment. Each unit module can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit module can be embedded in or independent of the processor in the computer device in hardware form, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above unit modules.

[0189] Example 3

[0190] like Figure 3 The diagram illustrates a terminal device disclosed in this embodiment, comprising a transmitter, a receiver, a memory, and a processor. The transmitter transmits instructions and data, the receiver receives instructions and data, the memory stores computer-executed instructions, and the processor executes the computer-executed instructions stored in the memory to implement the method described in Embodiment 1 above.

[0191] It should be noted that the above memory can be independent or integrated with the processor. When the memory is independently arranged, the terminal device further includes a bus for connecting the memory and the processor.

[0192] The above merely describes the preferred embodiments of the present application and is not intended to limit the protection scope of the present application. Any equivalent structure transformation made under the inventive concept of the present application, or direct / indirect application in other related technical fields, is included in the protection scope of the present application.

Claims

1. An image-based foreign object and target anomaly detection method, characterized by, The method comprises the following steps: Step 1, obtaining a reference image and a to-be-detected image, and performing noise reduction processing on the reference image and the to-be-detected image; Step 2, aligning the to-be-detected image with the reference image to obtain a mapping matrix of the to-be-detected image and the reference image; Step 3, comprising obtaining a preliminary foreign matter subgraph and obtaining a target reference subgraph: The process of obtaining the preliminary foreign matter subgraph is: respectively performing feature extraction on the to-be-detected image and the reference image to obtain a feature map atlas of the to-be-detected image and a feature map atlas of the reference image, and preliminarily detecting a foreign matter region coordinate in the to-be-detected image by using the difference between feature points in the two feature map atlases, and then segmenting the to-be-detected image based on the foreign matter region coordinate to obtain a preliminary foreign matter subgraph; The process of obtaining the target reference subgraph is: obtaining a target region coordinate by performing target detection on the reference image or directly anchoring the target, and then segmenting the reference image based on the target region coordinate to obtain a target reference subgraph; Step 4, comprising foreign matter detection and target anomaly detection: The process of foreign matter detection is: mapping the foreign matter region coordinate to the reference image based on the mapping matrix, segmenting the reference image based on the mapped foreign matter region coordinate to obtain a reference subgraph, and then forming a true-false identification image pair by combining the reference subgraph with the corresponding preliminary foreign matter subgraph, and performing brightness consistency processing; The process of target anomaly detection is: mapping the target region coordinate to the to-be-detected image based on the mapping matrix, segmenting the to-be-detected image based on the mapped target region coordinate to obtain a target suspected anomaly subgraph, and then forming a true-false identification image pair by combining the target suspected anomaly subgraph with the corresponding target reference subgraph, and performing brightness consistency processing; Step 5, performing cascaded true-false identification on the true-false identification image pair to complete foreign matter or target anomaly detection in the to-be-detected image, and the cascaded true-false identification of the true-false identification image pair specifically comprises: Based on the gray image, the illumination brightness difference features of the reference sub-image and the initial detection foreign matter sub-image in the genuine identification image pair, or the target suspected abnormal sub-image and the target reference sub-image are extracted , contrast difference features , target contour difference features ; Based on illumination intensity difference features , contrast difference features , target profile difference features , and calculating a similarity distance measure ; determining whether the first foreign object threshold or the first target abnormality threshold is lower: if so, determining that the foreign object in the current preliminary foreign object subgraph is a real foreign object or determining that the current target suspected abnormality subgraph is a target abnormality, otherwise determining that the foreign object in the current preliminary foreign object subgraph is a false foreign object or determining that the current target suspected abnormality subgraph is a target suspected abnormality; Constructing a feature classification extractor based on a neural network, extracting features of the true-false identification image pair determined as a false foreign matter or a target suspected anomaly by using the trained feature classification extractor, then evaluating the similarity of the extracted features to obtain a similarity evaluation metric; Judging whether the similarity evaluation metric is lower than a second foreign matter threshold or a second target anomaly threshold: if yes, determining that the false foreign matter is a real foreign matter or the target anomaly.

2. The image-based foreign object and target anomaly detection method of claim 1, wherein, Step 2 specifically comprises: Step 201, respectively extracting key points in the reference image and the to-be-detected image, and extracting feature vectors of the key points; Step 202, performing feature matching based on the feature vectors of the key points in the reference image and the to-be-detected image to obtain a plurality of key point pairs; Step 203, performing loop iteration deletion on outlier points in the key point pairs until a preset iteration termination condition is met; Step 204, performing coordinate mapping based on the remaining key point pairs to obtain a mapping matrix between the to-be-detected image and the reference image.

3. The image-based foreign object and target anomaly detection method of claim 2, wherein, In step 201, key points in the reference image and the image to be detected are extracted by constructing a multi-layer pyramid image, so that the feature vector of the key point has scale invariance; For any key point, The feature vector extraction process is specifically as follows: obtaining a scale image for key point extraction in a multi-layer pyramid image, and extracting a pixel pair centered on a key point in the scale image a neighborhood, and randomly obtaining a pixel pair in the neighborhood For any pair of pixels ( , Find the points in the given information respectively. The sum of the grayscale pixel values ​​in the neighborhood is taken as the pixel value of that point; A feature descriptor is obtained using the sum of pixel values in the neighborhood of the two points of the pixel pair To: wherein is a neighborhood pixel value of the pixel point , i.e. is a neighborhood pixel value of the pixel point , i.e.: wherein is a pixel value of the point ; Finally, the feature descriptor of the pixel point pair is concatenated, i.e. the feature vector of the key point is obtained, which is as follows: The feature descriptor of the pixel point pair is concatenated, i.e. the feature vector of the key point is obtained, which is as follows: wherein, is a feature vector for the keypoint.

4. The image-based foreign object and target anomaly detection method of claim 3, wherein, In random selection After the pixel point pairs are selected, the pixel point coordinates of the pixel point pair set are converted into the following matrix form: wherein, is a matrix of pixel coordinates for the set of pixel pairs; Then, the principal direction of the key point is used to obtain a rotation matrix, which is: wherein, is a rotation matrix, is a principal direction of the key point; Then, the pixel point coordinate matrix of the pixel point pair set is rotated by using a rotation matrix to obtain a rotated pixel point pair set matrix, which is as follows: wherein, is a set of rotated pixel pairs matrix, is a matrix multiplication; Finally, the feature vector of the key point is calculated based on the pixel pair set matrix , that is: At this point, the feature vector of the key point has rotation invariance.

5. The image-based foreign object and target anomaly detection method according to claim 2 or 3 or 4, characterized in that, Step 203 specifically includes: First, the first derivative of the keypoint pair in the x, y direction is calculated , ; Then, the mean and standard deviation of the first derivative of all keypoint pairs in the x direction , , and the mean and standard deviation of the first derivative of all keypoint pairs in the y direction , ; Finally, the discrete points are filtered in the x direction and the y direction respectively, and the iterative deletion is performed in cycles; The process of filtering the discrete points in the x direction and the iterative deletion in cycles is as follows: Step X1, calculating the first derivative series of the key point pair in the x direction the dispersion series of the mean value of the series is ;​ Step X2, calculate the mean of the series of x-dispersion values for all keypoint pairs and the standard deviation ; Step X3, the mean value is calculated and the standard deviation The lower and upper threshold values for the series of dispersion values of the pairs of key points in the x direction are calculated as: wherein, is a lower limit threshold value for the dispersion series of keypoint pairs in the x direction, is an upper limit threshold value for the dispersion series of keypoint pairs in the x direction; Step X4, retain those that satisfy the requirements. After removing other keypoint pairs, calculate the mean of the dispersion sequence of the remaining keypoint pairs in the x-direction. and standard deviation and judge and Is the difference less than the set threshold? If yes, the iterative deletion of the discrete points in the x direction is ended, and the remaining key point pairs are output; Otherwise, let 、 Then, go back to step X3. The process of filtering the discrete points in the y direction and the iterative deletion in cycles is as follows: Step Y1, get all key point pairs after iterative deletion of discrete points in x direction, then calculate the first derivative series of key point pairs in y direction the dispersion series of relative to the mean value of the series ;​ Step Y2, calculate the mean of the y-discreteness series of all keypoint pairs and the standard deviation ; Step Y3, the lower and upper threshold values of the dispersion series of the key point pairs in the y direction are calculated according to the mean value and the standard deviation The lower and upper threshold values of the dispersion series of the key point pairs in the y direction are calculated according to the mean value wherein, is a lower limit threshold value for the sequence of dispersion of the keypoint pair in the y direction, is an upper limit threshold value for the sequence of dispersion of the keypoint pair in the y direction; Step Y4, keep the key point pairs that satisfy and delete the other key point pairs, then calculate the mean value and standard deviation of the y-discreteness series of all the remaining key point pairs and judge whether the difference between and is less than the set threshold value ​ If yes, the iterative deletion of the discrete points in the y direction is ended, and the remaining key point pairs are output; Otherwise, let 、 After that, the process returns to step Y3.

6. The image-based foreign object and target anomaly detection method according to claim 1 or 2 or 3 or 4, characterized in that, In step 3, the acquisition process of the initial detection of the foreign matter subgraph specifically includes: Step 301, on the basis of the mapping matrix, four vertex coordinates of a common target region in the reference image and the image to be detected are extracted, and the vertex coordinates are used to cut the reference image and the image to be detected respectively, to obtain the aligned reference image and the image to be detected; Step 302, after the reference image and the image to be detected are aligned, the gabor features of the reference image and the gabor features of the image to be detected are extracted respectively, as the feature map atlas of the reference image and the image to be detected respectively; Step 303, the sift features of the feature map atlas of the reference image and the image to be detected are extracted respectively, to obtain the sift feature map atlas of the reference image and the image to be detected, and the sift feature map atlas of the reference image and the image to be detected is normalized; Step 304, the normalized two sift feature maps are compared with each other to obtain the similarity of each feature point in the two sift feature maps ; Step 305, a pure black image with the same size as the image to be detected is constructed, the points with feature similarity less than a set threshold in the sift feature map of the image to be detected are filtered out, the coordinates of all the filtered points are determined, and the pixel values of the corresponding coordinate positions in the pure black image are set to white, to obtain a black and white binary image; Step 306, the obtained black and white binary image is subjected to an erosion and expansion operation, the white pixels are connected into one or more regions, so that the foreign matter region coordinates in the image to be detected are obtained, and the initial detection of the foreign matter subgraph is obtained based on the foreign matter region coordinates in the image to be detected.

7. The image-based foreign object and target anomaly detection method according to claim 1 or 2 or 3 or 4, characterized in that, In step 4, the process of brightness consistency processing is as follows: For a set of true and false identification image pairs, convert the initial inspection foreign matter sub-graph and the reference sub-graph, or the target suspected abnormal sub-graph and the target reference sub-graph into gray scale images, and obtain the gray scale mean value of the gray scale image corresponding to the initial inspection foreign matter sub-graph or the target suspected abnormal sub-graph and the normalized brightness histogram , s, and the mean value of the gray scale image corresponding to the reference sub-graph and the normalized brightness histogram ; According to the normalized brightness histogram to calculate the difference distance , to ; Setting a threshold , , and ; When the normalized histogram of the initial foreign matter subgraph or the target suspected abnormality subgraph and its reference subgraph is calculated first , and the cumulative distribution function of the histogram , is calculated , so as to obtain the mapping function , and finally the initial foreign matter subgraph or the target suspected abnormality subgraph is subjected to brightness consistency processing according to the mapping function, that is , wherein is the initial foreign matter subgraph or the target suspected abnormality subgraph before brightness consistency processing, is the initial foreign matter subgraph or the target suspected abnormality subgraph after brightness consistency processing, is the coordinate of a pixel; when First, based on the mean of the grayscale image pairs... , Calculate the brightness consistency processing coefficient , ,for , Then, based on the brightness consistency coefficient... , Brightness consistency processing is performed on the initial foreign object sub-image or the sub-image of the suspected target anomaly, that is: .

8. An image-based foreign object and target anomaly detection system, characterized by, The detection system comprises the method according to any one of claims 1 to 7, An image preprocessing unit is configured to acquire a reference image and an image to be detected, and to perform noise reduction processing on the reference image and the image to be detected; An image alignment unit is configured to align the image to be detected with the reference image, to obtain a mapping matrix of the image to be detected and the reference image; An image preprocessing unit is configured to acquire a reference image and an image to be detected, and to perform noise reduction processing on the reference image and the image to be detected; The foreign matter segmentation unit is configured to respectively perform feature extraction on the to-be-detected image and the reference image, to obtain a feature map atlas of the to-be-detected image and a feature map atlas of the reference image, and to preliminarily detect a foreign matter region coordinate in the to-be-detected image by using a difference between feature points in the two feature map atlases, and then to segment the to-be-detected image based on the foreign matter region coordinate to obtain a preliminarily detected foreign matter sub-image; The target anomaly segmentation unit is configured to perform target detection on the reference image or directly anchor a target to obtain a target region coordinate, and then to segment the reference image based on the target region coordinate to obtain a target reference sub-image; The brightness processing unit is configured to map the foreign matter region coordinate to the reference image according to the mapping matrix, or to map the target region coordinate to the to-be-detected image according to the mapping matrix, and to segment the reference image based on the mapped coordinate to obtain a reference sub-image or to segment the to-be-detected image based on the mapped coordinate to obtain a target suspected anomaly sub-image, and then to form a true-false identification image pair by combining the reference sub-image with the preliminarily detected foreign matter sub-image or by combining the target suspected anomaly sub-image with the target reference sub-image, and to perform brightness consistency processing; The cascaded true-false identification unit is configured to perform cascaded true-false identification on the true-false identification image pair, to complete foreign matter or target anomaly detection in the to-be-detected image.

9. A terminal device, comprising: The terminal device is provided with: a memory configured to store a program; a processor configured to execute the program stored in the memory, and when the program is executed, the processor is configured to execute the method in any one of claims 1 to 7.

Images