Image-based foreign matter and target anomaly detection method and system, and terminal equipment

Abstract

The invention discloses an image-based foreign matter and target anomaly detection method and system and terminal equipment. The method comprises the following steps: aiming at foreign matter detection, extracting feature maps of a to-be-detected image and a reference image, initially detecting foreign matter area coordinates in the to-be-detected image by utilizing the difference of the two feature maps, and segmenting to obtain initially detected foreign matter sub-maps; the coordinates of the foreign matter area are mapped to the reference image to obtain a reference sub-image, and the reference sub-image and the initially detected foreign matter sub-image form a true and false identification image pair; for target anomaly detection, segmenting the reference image to obtain a target reference sub-image, mapping a target area coordinate to the to-be-detected image to obtain a target suspected anomaly sub-image, and forming an authenticity identification image pair with the target suspected anomaly sub-image; and performing cascade authenticity identification on the authenticity identification image pair to complete foreign matter or target anomaly detection in the to-be-detected image. The method is applied to the field of image processing and mode recognition, and can quickly and accurately detect whether foreign matters and targets in the to-be-detected image are abnormal or not.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, target detection and recognition, and in particular to an image-based foreign body and target anomaly detection method, system and terminal equipment. Background Art

[0002] In recent years, the rapid development of deep learning has achieved significant breakthroughs in all aspects of computer vision, especially in the field of object detection. As a key aspect of computer vision, object detection aims to detect objects in images and is a fundamental element of tasks such as object recognition, tracking, and object segmentation.

[0003] Foreign body detection and target anomaly detection technologies are often used in inspection tasks in fixed scenes such as pipelines. They belong to the field of target detection. Their detection targets are usually divided into two categories. One is foreign matter, that is, foreign objects existing in the scene. Its notable feature is that the type and shape of the object are unknown. The other is target anomaly, that is, target state abnormality, that is, the target object that is the focus of attention in the inspection scene has an abnormal state, such as cracks, damage, shedding, color, anti-counterfeiting patterns and other unforeseen defects and their forms on the surface.

[0004] Currently available deep learning models for target detection in situations where the type and morphology of foreign objects and target anomalies are unknown have significant shortcomings. For example, the accuracy of deep learning algorithms is highly dependent on the quality and quantity of the training dataset. Collecting samples of foreign objects and target anomalies is particularly difficult. Given the vast variety of objects in the world and their diverse postures, any object that is not part of the scene can be considered a foreign object. Therefore, it is impossible to collect all foreign objects as training samples. The same is true for target anomalies, such as cracks, skew, peeling, ablation, or forgery, which can cause unpredictable defects such as color changes and position changes. Furthermore, the accuracy of deep learning models suffers from the well-known long-tail effect, meaning that the recognition rate of a small number of objects is very low. Despite various academic improvements to deep models based on long-tail datasets, their detection accuracy still falls far short of the stringent requirements for practical applications. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the prior art, the present invention provides an image-based foreign object and target anomaly detection method, system and terminal device, which can quickly and accurately detect foreign objects and target anomalies in the image to be detected.

[0006] To achieve the above object, the present invention provides an image-based foreign body and target anomaly detection method, comprising the following steps: Step 1: Acquire a reference image and an image to be detected, and perform noise reduction processing on the reference image and the image to be detected; Step 2: aligning the image to be detected with the reference image to obtain a mapping matrix between the image to be detected and the reference image; Step 3 includes obtaining the initial foreign body sub-image and the target reference sub-image: The process of obtaining the preliminary foreign body sub-graph is as follows: performing feature extraction on the image to be detected and the reference image respectively to obtain a feature map of the image to be detected and a feature map of the reference image, and using the difference between the feature points in the two feature maps to preliminarily detect the coordinates of the foreign body region in the image to be detected, and then segmenting the image to be detected based on the foreign body region coordinates to obtain the preliminary foreign body sub-graph; The target reference sub-image is obtained by performing target detection on the reference image or directly anchoring the target to obtain target region coordinates, and then performing segmentation on the reference image based on the target region coordinates to obtain the target reference sub-image; Step 4, including foreign object detection and target anomaly detection: The foreign body detection process comprises: mapping the coordinates of the foreign body region onto the reference image based on the mapping matrix, segmenting the reference image based on the mapped foreign body region coordinates to obtain a reference sub-image, then combining the reference sub-image with the corresponding foreign body sub-image of the initial inspection to form an authenticity identification image pair, and performing brightness consistency processing; The target anomaly detection process comprises: mapping the target region coordinates onto the image to be detected based on the mapping matrix, segmenting the image to be detected based on the mapped target region coordinates to obtain a target suspected anomaly sub-image, then combining the target suspected anomaly sub-image with the corresponding target reference sub-image to form an authenticity identification image pair, and performing brightness consistency processing; Step 5: Perform cascade authentication on the authenticity authentication image pair to complete the detection of foreign objects or target anomalies in the image to be detected.

[0007] To achieve the above objectives, the present invention further provides an image-based foreign body and target anomaly detection system, using the above method, the detection system comprises: An image preprocessing unit, configured to obtain a reference image and an image to be detected, and perform noise reduction processing on the reference image and the image to be detected; an image alignment unit, configured to align the image to be detected with the reference image to obtain a mapping matrix between the image to be detected and the reference image; a foreign body segmentation unit, configured to perform feature extraction on the image to be detected and the reference image respectively, to obtain a feature map atlas of the image to be detected and a feature map atlas of the reference image, and to preliminarily detect the coordinates of the foreign body region in the image to be detected by using the difference between the feature points in the two feature maps, and then to segment the image to be detected based on the foreign body region coordinates to obtain a preliminarily detected foreign body sub-map; a target anomaly segmentation unit, configured to perform target detection on the reference image or directly anchor the target to obtain target region coordinates, and then segment the reference image based on the target region coordinates to obtain a target reference sub-image; a brightness processing unit, configured to map the coordinates of the foreign body region onto the reference image according to the mapping matrix, or map the coordinates of the target region onto the image to be detected according to the mapping matrix, and segment the reference image to obtain a reference sub-image or segment the image to be detected to obtain a target suspected abnormal sub-image based on the mapped coordinates, and then form an authenticity identification image pair with the reference sub-image and the corresponding foreign body sub-image of the initial inspection, or with the target suspected abnormal sub-image and the corresponding target reference sub-image, and perform brightness consistency processing; The cascaded authenticity identification unit is used to perform cascaded authenticity identification on the authenticity identification image pair to complete the detection of foreign matter or target anomaly in the image to be detected.

[0008] To achieve the above object, the present invention further provides a terminal device, wherein the terminal device is provided with: Memory, used to store programs; The processor is configured to execute the program stored in the memory. When the program is executed, the processor is configured to execute the method described above.

[0009] Compared with the prior art, the present invention has the following beneficial technical effects: 1. The present invention detects the presence of foreign objects or abnormalities in a scene through image feature comparison and foreign object authenticity identification. It can not only quickly and accurately detect foreign objects and target anomalies in the image to be detected, but also deeply overcomes the problem that the accuracy of neural network model algorithms in foreign object and target anomaly detection is heavily dependent on the conditions such as the complete variety, rich posture and balanced number of foreign object samples and target defect anomaly samples. At the same time, its adaptability and high robustness are completely unlimited. It is not limited to the unknown characteristics of foreign object type, posture, number, etc. 2. The present invention has been integrated into inspection equipment such as robots and drones equipped with cameras, and can also be deployed on product defect detection equipment on production lines, and has a wide range of applications in reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0011] Figure 1 Flowchart of the method for detecting foreign matter and target anomalies based on an image in Example 1 of the present invention; Figure 2 4 is a block diagram of the structure of the image-based foreign object and target anomaly detection system in Example 2 of the present invention; Figure 3 This is a structural block diagram of the terminal device in Example 3 of the present invention.

[0012] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0013] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0014] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0015] Example 1 This embodiment discloses an image-based foreign body and target anomaly detection method, which can not only quickly and accurately detect foreign bodies and target anomalies in the image to be detected, but also deeply overcomes the defect that the accuracy of the neural network model algorithm in foreign body and target anomaly detection is heavily dependent on the complete variety, rich posture and balanced number of foreign body samples. At the same time, its adaptability and high robustness are completely not limited to the unknown characteristics of the type, posture, and number of foreign bodies and target anomalies. The detection method in this embodiment can be applied to scenarios such as maintenance of civil equipment and defect detection of industrial products. It can be integrated into inspection equipment such as robots and drones equipped with cameras, and can also be deployed on product defect detection equipment on production lines. For example, inspection and detection of important targets in underground pipelines and pipe networks in the field of civil equipment, inspection and detection of power line and power grid safety, defect detection of products in the industrial field, etc.

[0016] refer to Figure 1 In this embodiment, the image-based foreign body and target anomaly detection method specifically includes the following steps: Step 1: Obtain a reference image and an image to be detected, and perform noise reduction on the reference image and the image to be detected; Step 2: Align the image to be detected with the reference image to obtain a mapping matrix between the image to be detected and the reference image; Step 3 includes obtaining the initial foreign body sub-image and the target reference sub-image: The process of obtaining the preliminary foreign body sub-image is as follows: feature extraction is performed on the image to be detected and the reference image respectively to obtain the feature map of the image to be detected and the feature map of the reference image. The coordinates of the foreign body area in the image to be detected are preliminarily detected by using the difference between the feature points in the two feature maps. The preliminary foreign body sub-image is then segmented on the image to be detected based on the foreign body area coordinates. The target reference sub-image is obtained by performing target detection on the reference image or directly anchoring the target to obtain the target region coordinates, and then performing segmentation on the reference image based on the target region coordinates to obtain the target reference sub-image. Step 4, including foreign object detection and target anomaly detection: The process of foreign body detection is as follows: the coordinates of the foreign body area are mapped to the reference image based on the mapping matrix, and the reference sub-image is obtained by segmenting the reference image based on the mapped foreign body area coordinates. The reference sub-image and the corresponding initial foreign body sub-image are then combined into an authenticity identification image pair, and brightness consistency processing is performed. The target anomaly detection process is as follows: mapping the target region coordinates to the image to be detected based on the mapping matrix, segmenting the image to be detected based on the mapped target region coordinates to obtain a target suspected anomaly sub-image, then combining the target suspected anomaly sub-image with the corresponding target reference sub-image to form an authenticity identification image pair, and performing brightness consistency processing; Step 5: Perform cascade authentication on the authenticity authentication image pair to complete the detection of foreign objects or target anomalies in the image to be detected.

[0017] In this embodiment, the reference image refers to an image captured by a camera at a specific monitoring point at a specific time. This image generally shows no foreign objects or abnormalities in the scene, except for the target of interest. The reference image can be replaced by any image captured at that monitoring point. The image to be inspected refers to an image captured by the same camera at the same monitoring point. The image to be inspected and determined for foreign objects or abnormalities in the scene is then used to transmit the inspection results back to the visualization interface, allowing personnel or robots to perform maintenance at the monitoring point.

[0018] In the specific implementation process of step 1, a 2D Gaussian filter can be used to reduce the noise of the reference image and the image to be detected. It is worth noting that in the specific application process, mean filtering, morphological denoising, etc. can also be used to reduce the noise of the reference image and the image to be detected.

[0019] In the specific application process, although the reference image and the image to be detected are collected by the same camera, because the reference image and the image to be detected are acquired at different times, the absolute position of the camera when the two images are collected may not be exactly the same, resulting in different position values ​​of the objects in the two image scenes. Therefore, it is necessary to align the two images before performing foreign object detection. In the specific implementation process, the image to be detected is aligned with the reference image, and the mapping matrix between the image to be detected and the reference image is obtained. The specific steps include: Step 201: extract key points from the reference image and the image to be detected, and obtain feature vectors of the key points; Step 202, performing feature matching based on the feature vectors of the key points in the reference image and the image to be detected to obtain a plurality of key point pairs; Step 203, iteratively deleting outliers in the key point pairs until a preset iteration termination condition is met; Step 204 : performing coordinate mapping based on the remaining key point pairs to obtain a mapping matrix between the image to be detected and the reference image.

[0020] In the specific implementation process, to ensure that the feature vectors of key points are scale-invariant, this embodiment extracts key points by constructing a multi-layer pyramid image. Specifically, the original reference image and the image to be detected are scaled to different scales, such as 1 / 1.2 or 1 / 1.5 of the original size. A pixel neighborhood on a certain layer of the pyramid image is then determined to be a key point. If not, the next pixel is determined to be a key point. Otherwise, the key point's position, correlation value, neighborhood size, and image scale are recorded. The key point's position and neighborhood are then transformed onto images at other scales to determine whether the point is a key point on the current scale. If it is, the key point's position and neighborhood are further transformed onto images at other scales to determine whether it is a key point. Otherwise, in the constructed multi-layer pyramid image, if the current key point is not a key point after scale transformation at any scale (layer) of the image, it is not a key point, and key point determination is continued at the next pixel. If all the current pyramid images identify the point as a keypoint, the extreme value is taken from the correlation values ​​recorded for that point at each scale, and the position and scale of the keypoint are recorded. This process is repeated until all keypoints, their positions, and scales are extracted. Finally, based on the scale information recorded for each keypoint, the scale with the most occurrences is found. The keypoint at that scale is the final keypoint to be extracted, and the image at that scale is the image with the best feature vector.

[0021] In the specific application process, the method of extracting key points in a certain level image of the multi-layer pyramid image is as follows: ; in, 、 is the coordinate position of the image pixel, for point The neighborhood size at is usually an odd number; Pixel Distance pixel The weight of is related to the relative distance between the two pixels; For the selected point The pixel value at for point The pixel value at for point with dot The absolute difference of pixel values, is the number of key points in the image, Is the threshold for whether it is a key point.

[0022] After determining the scale image with the best feature vector extraction, the process of extracting key points on the scale image is as follows: First, extract the key point centered on the scale image. neighborhood, and randomly obtain Pixel pairing; Then, for any pair of pixels ( , ) in the points The sum of the grayscale pixel values ​​in the neighborhood is used as the pixel value of the point, thereby reducing noise interference; Then, a feature descriptor is obtained by using the sum of the pixel values ​​in the neighborhood of the two points in the pixel pair ,for:

[0023] in, Pixel The neighborhood pixel value of Pixel The neighborhood pixel value of , that is: ; in, for point Pixel value of Finally, The feature descriptors of the pixel pairs are connected in series to obtain the feature vector of the key point, which is: ; in, is the feature vector of the key point; is the number of pixel pairs, is the first pixel in the feature vector Dimensional features.

[0024] As a preferred embodiment, in order to make the feature vector of the key point have rotation invariance, After matching the pixels, the following operations are performed: First, randomly select After pairing the pixels, the pixel coordinates of the pixel pair set are converted into the following matrix form:

[0025] in, is the pixel coordinate matrix of the pixel pair set; Then, the rotation matrix is ​​obtained using the main direction of the key point, which is:

[0026] in, is the rotation matrix, is the main direction of the key point; Then, the pixel coordinate matrix of the pixel pair set is transformed using the rotation matrix Perform the rotation and get the rotated pixel pair matrix, which is:

[0027] in, is the matrix of pixel pairs after rotation, is matrix multiplication; Finally, based on the pixel pair matrix The pixel pairs in Calculate the feature vector of the pixel, that is: .

[0028] At this point, the feature vector of the key point can be made rotation invariant.

[0029] In the specific implementation process, when the grayscale of a key point deviates from the centroid of the neighborhood, the main direction of the key point can be obtained through the vector from the key point to the centroid. , and its calculation process is as follows: First, calculate the geometric moment of the image, which is ,in, is a non-negative number, The sum is the order of the geometric moment, such as is the second-order moment of the image; Then, the centroid of the neighborhood image is obtained using the 0th-order moment and the 1st-order moment, which is ; Finally, the main direction of the key point can be calculated .

[0030] In the specific implementation process, the process of feature matching the feature vectors of key points is as follows: First, calculate the Hamming distance between the feature vectors of two key points. ; in, 、 are the feature vectors of a key point pair in the reference image and the image to be detected, is the Hamming distance between two eigenvectors, For or calculate; The smaller the value of , the higher the similarity between the two feature vectors, and vice versa. The value meets the agreed threshold When , the key points in the reference image and the image to be detected are determined and The matching is successful, thus obtaining the coordinate values ​​of the corresponding key points in the two images ,for:

[0031] in, are the coordinates of the key points that are successfully matched in the reference image and the image to be detected, respectively.

[0032] To ensure the accuracy of subsequent image alignment, it is necessary to further screen the correctly matched key point pairs and delete those outlier key points. In this embodiment, the second-order derivatives of the paired key points in the x- and y-directions, as well as their variances and means, should all be within a reasonable range as a benchmark, and the key point pairs are screened through a loop iteration. The specific process is as follows: First, calculate the first-order derivative of the key point pair in the x and y directions as: ; in, 、 is the value of the two key points in the key point pair in the x direction, is the first-order derivative of the key point pair in the x direction, 、 is the value of the two key points in the key point pair in the y direction, is the first-order derivative of the key point pair in the y direction, is the number of key point pairs; Then, calculate the mean and standard deviation of the first-order derivatives of all key point pairs in the x and y directions as follows:

[0033] ; in, is the mean of the first-order derivatives of all key point pairs in the x direction, is the standard deviation of the first-order derivatives of all key point pairs in the x direction, is the mean of the first-order derivatives of all key point pairs in the y direction, is the standard deviation of the first-order derivative of all key point pairs in the y direction; Finally, the discrete points are filtered in the x-direction and the y-direction and deleted iteratively.

[0034] In this embodiment, the process of screening discrete points in the x direction and iteratively deleting them is as follows: Step X1, calculate the first-order derivative series of the key point pair in the x direction Relative to the mean of the series The discrete degree series ,for:

[0035] Step X2: Calculate the mean of the discrete degree series of all key point pairs in the x direction and standard deviation ,for: .

[0036] Step X3, based on the mean and standard deviation Calculate the lower threshold of the discrete degree series of key points in the x direction With upper threshold ,for: ; .

[0037] Step X4, keep satisfied After deleting the other key point pairs, calculate the mean of the discrete degree series of all remaining key point pairs in the x direction and standard deviation , and judge and Is the difference less than the set threshold? If so, end the iterative deletion of discrete points in the x-direction and output the remaining key point pairs; Otherwise, let 、 Then, return to step X3.

[0038] In this embodiment, the process of screening discrete points in the y direction and iteratively deleting them is as follows: Step Y1: Get all key point pairs after iterative deletion of discrete points in the x direction, and then calculate the first-order derivative series of the key point pairs in the y direction. Relative to the mean of the series The discrete degree series ,for:

[0039] Step Y2: Calculate the mean of the discrete degree series of all key point pairs in the y direction and standard deviation ,for: ; in, The number of key point pairs remaining after iterative deletion of discrete points in the x direction; Step Y3, based on the mean and standard deviation Calculate the lower threshold of the discrete degree series of key points in the y direction With upper threshold ,for: ; .

[0040] Step Y4, retain the content After deleting the other key point pairs, calculate the mean of the discrete degree series of all remaining key point pairs in the y direction and standard deviation , and judge and Is the difference less than the set threshold? If so, end the iterative deletion of discrete points in the y direction and output the remaining key point pairs; Otherwise, let 、 Then, return to step Y3.

[0041] It is worth noting that, in a specific application process, the discrete points in the y direction may be iteratively deleted first, and then the discrete points in the x direction may be iteratively deleted.

[0042] After completing the iterative deletion of discrete points in the x- and y-directions, coordinate mapping can be performed based on the remaining key point pairs. The mapping matrix between the image to be detected and the reference image is obtained by using the coordinates of the key points in the image to be detected and their paired coordinates on the reference image, which is: ; in, The x and y in are the coordinates of the key points on the reference image. is the scaling factor, is the mapping matrix, in , is the coordinate of the key point on the image to be detected; Using the mapping matrix, the pixel coordinates on the image to be detected can be mapped to the corresponding positions on the reference image, thus completing the alignment of the reference image and the image to be detected.

[0043] In the specific implementation process of step 3, the process of extracting the feature map of the image to be detected and the feature map of the reference image is as follows: Step 301: extract the coordinates of four vertices of the common target area in the two images, and then use the vertex coordinates to intercept the reference image and the image to be detected respectively to obtain aligned reference image and image to be detected; Step 302: After the reference image and the image to be detected are aligned, Gabor features of the reference image and Gabor features of the image to be detected are extracted, and the features are used as feature maps of the reference image and the image to be detected respectively; Step 303: extract SIFT features from the feature maps of the reference image and the image to be detected respectively to obtain the SIFT feature maps of the reference image and the image to be detected, and perform feature normalization on the SIFT feature maps of the reference image and the image to be detected; Step 304: calculate the feature similarity of each feature point in the two normalized sift feature maps to obtain the similarity of each feature point in the two sift feature maps. ,for: ; in, 、 They are the sift feature maps of the extracted reference image and the aligned image to be detected respectively; The range is If its value is closer to 0, the similarity of the feature vector is very poor, and it can be judged that the point is a point on the foreign object; if it is closer to 1, the similarity of the corresponding points in the two images is very high, and it is a normal point; Step 305: construct a pure black image with the same size as the image to be detected, filter out points in the SIFT feature map of the image to be detected whose feature similarity is less than a set threshold, determine the coordinates of the filtered points, and set the pixel values ​​of the corresponding coordinate positions in the pure black image to 225 (i.e., white) to obtain a black and white binary image. The set threshold value of the feature similarity is an empirical value and can be set to 0.8 to ensure that no missed detection occurs. In step 306, the obtained black and white binary image is subjected to an erosion and dilation operation, and then connected in series using a connected domain algorithm or a flooding algorithm to connect the white pixels into one or more regions, thereby preliminarily detecting the coordinates of the foreign body region in the image to be detected, and then segmenting the image to be detected based on the coordinates of the foreign body region to obtain a preliminarily detected foreign body sub-image.

[0044] Since the reference image and the image to be detected are taken at different time periods, if the ambient light is inconsistent or the fill light is different, the illumination in the two images will be inconsistent, which will lead to false alarms of foreign objects or target anomalies. Therefore, it is necessary to perform image brightness consistency processing on the segmented foreign object sub-image or target suspected abnormal sub-image before authenticity verification. The specific process of brightness consistency processing in this embodiment is as follows: Convert the authenticity identification image pair into grayscale image pair and calculate their grayscale mean respectively 、 and normalized brightness histogram , ),for: ; in, 、 are respectively the histograms of the initially detected foreign body sub-image or the target suspected abnormal sub-image and its reference sub-image or the target reference sub-image in the authenticity identification image pair; Calculate the difference distance based on the normalized brightness histogram ,for: ; The value range is [0,1]. When the value is 0, it means that the two images are completely consistent. When the value exceeds a certain value, it means that there is a significant brightness difference between the two images in the authenticity identification image pair, and brightness consistency processing is required.

[0045] Then according to With threshold 、 The brightness consistency processing is performed on the initial detection foreign body sub-image or the target suspected abnormal sub-image in the authenticity identification image pair based on the relationship between the two.

[0046] when When , the process of brightness consistency processing for the initial detection foreign body sub-image or the target suspected abnormal sub-image in the authenticity identification image pair is as follows: First, according to the histogram of the initial foreign body sub-image or the target suspected abnormal sub-image and its reference sub-image in the authenticity identification image, the normalized histogram of the initial foreign body sub-image or the target suspected abnormal sub-image and its reference sub-image is calculated. 、 ,for: ; in, 、 are the number of pixels in the initial foreign body sub-image or target suspected abnormal sub-image and its reference sub-image or target reference sub-image respectively; Then calculate the cumulative distribution function of the normalized histogram 、 ,for: .

[0047] Then, based on the cumulative distribution function, the mapping function of the brightness consistency between the initial detection foreign body sub-image or the target suspected abnormal sub-image in the authenticity identification image pair and its reference sub-image or the target reference sub-image is obtained. ,for:

[0048] Finally, according to the mapping function Perform brightness consistency processing on the initially detected foreign body sub-image or the target suspected abnormal sub-image, namely: ; in, This is the initial inspection of foreign body sub-images or target suspected abnormal sub-images before brightness consistency processing. This is the initial inspection foreign body sub-image or target suspected abnormal sub-image after brightness consistency processing. is the pixel coordinate.

[0049] when When , the process of brightness consistency processing for the initial detection foreign body sub-image or the target suspected abnormal sub-image in the authenticity identification image pair is as follows: First, the grayscale image mean of the true and false identification image pair 、 Calculate brightness consistency processing coefficient 、 ,for: ; .

[0050] Then, according to the brightness consistency processing coefficient 、 Perform brightness consistency processing on the initially detected foreign body sub-image or the target suspected abnormal sub-image, namely:

[0051] Due to the low threshold setting for feature similarity in step 305 and the large difference in lighting between the authenticity verification images, there will be a certain false alarm rate in the initial foreign object detection or target anomaly detection. Therefore, it is necessary to verify the authenticity of the initially detected foreign objects or target anomaly sub-images. To ensure a high detection rate, this embodiment uses a cascaded authenticity verification method in step 5 to verify the authenticity of the initially detected foreign objects or target anomalies. The specific implementation process includes two parts: authenticity verification based on multiple feature fusion and authenticity verification based on deep neural networks.

[0052] In the specific implementation process, the authenticity identification based on the fusion of multiple features specifically includes: The authenticity identification image pairs that have undergone brightness consistency processing are resized, that is, the reference sub-image or the target reference sub-image is scaled to the same size as the initial foreign body sub-image or the target suspected abnormal sub-image through bilinear interpolation; Extract the brightness difference features of the reference sub-image and the initial foreign body sub-image in the authenticity identification image based on the grayscale image, or the target reference sub-image and the target suspected abnormal sub-image , contrast difference characteristics , target contour difference characteristics ,for:

[0053] in, 、 It is the grayscale mean and variance of the initial inspection of the foreign body sub-image or the target suspected abnormal sub-image after brightness consistency processing, 、 is the grayscale mean and variance of the reference sub-image or target reference sub-image, is the covariance of the true and false identification image pair, is the number of pixels in the x-direction of the initial inspection foreign body sub-image or the target suspected abnormal sub-image, the reference sub-image or the target reference sub-image, is the number of pixels in the y direction of the initial detection foreign body sub-image or the target suspected abnormal sub-image, the reference sub-image or the target reference sub-image, is the reference subgraph or target reference subgraph, This is the initial inspection foreign body sub-image or target suspected abnormal sub-image after brightness consistency processing. 、 is the weight coefficient; Based on the light intensity difference feature vector , contrast difference feature vector , target contour difference feature vector , calculate the similarity distance metric ,for:

[0054] in" " is the vector dot product, The value range is between [0,1]. When the value is 1, the two images in the authenticity identification image pair are exactly the same. When the value is 0, the two images in the authenticity identification image pair are completely different images. judge Is it lower than the first foreign body threshold or the first target abnormality threshold: If so, the foreign body in the current initial inspection foreign body sub-image is determined to be a real foreign body or a target abnormality; otherwise, the foreign body in the current initial inspection foreign body sub-image is determined to be a false foreign body or the current initial inspection target abnormality sub-image is determined to be a suspected target abnormality.

[0055] After the authenticity verification based on multiple feature fusion, the authenticity verification images identified as fake foreign objects or suspected abnormal targets will undergo a secondary judgment, that is, the authenticity verification based on deep neural networks. Specifically: Authenticity identification based on deep neural networks uses a neural network to train a feature classification extractor. Currently, classic feature classifiers such as AlexNet, ResNet, Vision Transformer and other network models can be directly used to train feature extractors. You can also design a feature extraction and classification model to train it. A trained feature classification extractor is used to extract features from the authenticity identification image pairs determined to be pseudo-foreign objects or the authenticity identification image pairs of the target suspected to be abnormal, and then a similarity evaluation is performed on the extracted features. In this embodiment, the cosine distance is used as the similarity evaluation metric. The closer the metric value is to 0, the higher the credibility that the pseudo-foreign object is a real foreign object or the target is normal. Otherwise, the higher the credibility that it is a pseudo-foreign object or the target is abnormal. For example, when the similarity evaluation metric is less than the second foreign object threshold, the pseudo-foreign object is determined to be a real foreign object. When the similarity evaluation metric is less than the second target abnormality threshold, the target state is determined to be abnormal. At this point, the foreign object or target abnormality detection is completed, and the abnormal status of the foreign object or target can be output with high credibility.

[0056] It is worth noting that although this embodiment Figure 1 The steps in the diagram are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0057] Example 2 Based on the image-based foreign body and target anomaly detection method in Example 1, this embodiment discloses an image-based foreign body and target anomaly detection system, referring to Figure 2 The image-based foreign body and target anomaly detection system includes an image preprocessing unit, an image alignment unit, a foreign body segmentation unit, a target anomaly segmentation unit, a brightness processing unit, and a cascaded authenticity identification unit. Specifically: The image preprocessing unit is used to obtain the reference image and the image to be detected, and perform noise reduction processing on the reference image and the image to be detected; The image alignment unit is used to align the image to be detected with the reference image to obtain a mapping matrix between the image to be detected and the reference image; The foreign body segmentation unit is used to extract features from the image to be detected and the reference image respectively, obtain a feature map of the image to be detected and a feature map of the reference image, and use the difference between the feature points in the two feature maps to preliminarily detect the coordinates of the foreign body area in the image to be detected, and then segment the image to be detected based on the foreign body area coordinates to obtain a preliminarily detected foreign body sub-map; The target anomaly segmentation unit is used to perform target detection on the reference image or directly anchor the target to obtain the target area coordinates, and then segment the reference image based on the target area coordinates to obtain the target reference sub-image; a brightness processing unit for mapping the coordinates of the foreign body region onto the reference image according to the mapping matrix, or mapping the coordinates of the target region onto the image to be detected according to the mapping matrix, and segmenting the reference image to obtain a reference sub-image based on the mapped coordinates, or segmenting the target suspected abnormal sub-image based on the image to be detected, and then combining the reference sub-image with the corresponding initial foreign body sub-image, or the target suspected abnormal sub-image with the corresponding target reference sub-image, to form an authenticity identification image pair, and perform brightness consistency processing; The cascaded authenticity identification unit is used to perform cascaded authenticity identification on the authenticity identification image pair, and complete the detection of foreign objects or target anomalies in the image to be detected.

[0058] In this embodiment, the specific operating processes and operating principles of the image preprocessing unit, image alignment unit, foreign object segmentation unit, brightness processing unit, and cascaded authenticity verification unit are the same as those in Example 1, and therefore will not be described in detail in this embodiment. Each unit module can be implemented in whole or in part through software, hardware, or a combination thereof. Each unit module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the corresponding operations of each of the above unit modules.

[0059] Example 3 like Figure 3 The terminal device disclosed in this embodiment includes a transmitter, a receiver, a memory, and a processor. The transmitter is used to send instructions and data, the receiver is used to receive instructions and data, the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions stored in the memory to implement the method in the above-mentioned embodiment 1.

[0060] It should be noted that the above memory can be independent or integrated with the processor. When the memory is independently provided, the terminal device further includes a bus for connecting the memory and the processor.

[0061] The above description is only a preferred embodiment of the present invention and does not limit the scope of protection of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the scope of protection of the present invention.

Claims

1. A method for detecting foreign objects and target anomalies based on an image, characterized in that: The steps include: Step 1: Acquire a reference image and an image to be detected, and perform noise reduction processing on the reference image and the image to be detected; Step 2: aligning the image to be detected with the reference image to obtain a mapping matrix between the image to be detected and the reference image; Step 3 includes obtaining the initial foreign body sub-image and the target reference sub-image: The process of obtaining the preliminary foreign body sub-graph is as follows: performing feature extraction on the image to be detected and the reference image respectively to obtain a feature map of the image to be detected and a feature map of the reference image, and using the difference between the feature points in the two feature maps to preliminarily detect the coordinates of the foreign body region in the image to be detected, and then segmenting the image to be detected based on the foreign body region coordinates to obtain the preliminary foreign body sub-graph; The target reference sub-image is obtained by performing target detection on the reference image or directly anchoring the target to obtain target region coordinates, and then performing segmentation on the reference image based on the target region coordinates to obtain the target reference sub-image; Step 4, including foreign object detection and target anomaly detection: The foreign body detection process comprises: mapping the coordinates of the foreign body region onto the reference image based on the mapping matrix, segmenting the reference image based on the mapped foreign body region coordinates to obtain a reference sub-image, then combining the reference sub-image with the corresponding foreign body sub-image of the initial inspection to form an authenticity identification image pair, and performing brightness consistency processing; The target anomaly detection process comprises: mapping the target region coordinates onto the image to be detected based on the mapping matrix, segmenting the image to be detected based on the mapped target region coordinates to obtain a target suspected anomaly sub-image, then combining the target suspected anomaly sub-image with the corresponding target reference sub-image to form an authenticity identification image pair, and performing brightness consistency processing; Step 5: Perform cascade authentication on the authenticity authentication image pair to complete the detection of foreign objects or target anomalies in the image to be detected.

2. The image-based foreign body and target anomaly detection method according to claim 1, characterized in that: Step 2 specifically includes: Step 201, extracting key points from the reference image and the image to be detected, and extracting feature vectors of the key points; Step 202, performing feature matching based on the feature vectors of the key points in the reference image and the image to be detected to obtain a plurality of key point pairs; Step 203, iteratively deleting outliers in the key point pairs until a preset iteration termination condition is met; Step 204 : performing coordinate mapping based on the remaining key point pairs to obtain a mapping matrix between the image to be detected and the reference image.

3. The image-based foreign body and target anomaly detection method according to claim 2, characterized in that: In step 201, key points in the reference image and the image to be detected are extracted by constructing a multi-layer pyramid image so that the feature vectors of the key points have scaling invariance; For any key point, The specific process of extracting its feature vector is as follows: Get the scale image for key point extraction in the multi-layer pyramid image, and extract the key point centered on the scale image neighborhood, and randomly obtain Pixel pairing; For any pair of pixels ( , ) in the points The sum of the grayscale pixel values ​​in the neighborhood is taken as the pixel value of the point; A feature descriptor is obtained by using the sum of the pixel values ​​in the neighborhood of two points in the pixel pair. ,for: in, Pixel The neighborhood pixel value of Pixel The neighborhood pixel value of , that is: in, for point Pixel value of Finally, The feature descriptors of the pixel pairs are connected in series to obtain the feature vector of the key point, which is: in, is the feature vector of the key point.

4. The image-based foreign body and target anomaly detection method according to claim 3, characterized in that: In random selection After pairing the pixels, the pixel coordinates of the pixel pair set are converted into the following matrix form: in, is the pixel coordinate matrix of the pixel pair set; Then, the rotation matrix is ​​obtained using the main direction of the key point, which is: in, is the rotation matrix, is the main direction of the key point; Then, the pixel coordinate matrix of the pixel pair set is transformed using the rotation matrix Perform the rotation and get the rotated pixel pair matrix, which is: in, is the matrix of pixel pairs after rotation, is matrix multiplication; Finally, based on the pixel pair matrix The pixel pairs in Calculate the feature vector of the key point, that is: At this point, the feature vector of the key point has rotation invariance.

5. The image-based foreign body and target anomaly detection method according to claim 2, 3 or 4, characterized in that: Step 203 specifically includes: First, calculate the first-order derivative of the key point pair in the x and y directions 、 ; Then, calculate the mean and standard deviation of the first-order derivatives in the x direction for all keypoint pairs 、 , as well as the mean and standard deviation of the first-order derivatives of all key point pairs in the y direction 、 ; Finally, the discrete points are screened in the x-direction and the y-direction and deleted iteratively; The process of screening discrete points in the x-direction and iteratively deleting them is as follows: Step X1, calculate the first-order derivative series of the key point pair in the x direction Relative to the mean of the series The discrete degree series ,for ; Step X2: Calculate the mean of the discrete degree series of all key point pairs in the x direction and standard deviation ; Step X3, based on the mean and standard deviation Calculate the lower and upper thresholds of the discrete degree series of key point pairs in the x direction as follows: in, is the lower limit threshold of the discrete degree series of key points in the x direction, is the upper threshold of the discrete degree series of key point pairs in the x direction; Step X4, keep satisfied After deleting the other key point pairs, calculate the mean of the discrete degree series of all remaining key point pairs in the x direction and standard deviation , and judge and Is the difference less than the set threshold? If so, end the iterative deletion of discrete points in the x-direction and output the remaining key point pairs; Otherwise, let 、 Then, return to step X3; The process of screening discrete points in the y direction and iteratively deleting them is as follows: Step Y1: Get all key point pairs after iterative deletion of discrete points in the x direction, and then calculate the first-order derivative series of the key point pairs in the y direction. Relative to the mean of the series The discrete degree series ; Step Y2: Calculate the mean of the discrete degree series of all key point pairs in the y direction and standard deviation ; Step Y3, based on the mean and standard deviation Calculate the lower and upper thresholds of the discrete degree series of key point pairs in the y direction as follows: in, is the lower threshold of the discrete degree series of key points in the y direction, is the upper threshold of the discrete degree series of key point pairs in the y direction; Step Y4, retain the content After deleting the other key point pairs, calculate the mean of the discrete degree series of all remaining key point pairs in the y direction and standard deviation , and judge and Is the difference less than the set threshold? If so, end the iterative deletion of discrete points in the y direction and output the remaining key point pairs; Otherwise, let 、 Then, return to step Y3.

6. The image-based foreign body and target anomaly detection method according to claim 1, 2, 3 or 4, characterized in that: In step 3, the process of obtaining the initial foreign body sub-graph specifically includes: Step 301: extracting the coordinates of four vertices of the common target area in the reference image and the image to be detected based on the mapping matrix, and using the vertex coordinates to intercept the reference image and the image to be detected respectively to obtain aligned reference image and image to be detected; Step 302: After the reference image and the image to be detected are aligned, Gabor features of the reference image and Gabor features of the image to be detected are extracted as feature maps of the reference image and the image to be detected, respectively. Step 303: extracting SIFT features from the feature maps of the reference image and the image to be detected respectively to obtain the SIFT feature maps of the reference image and the image to be detected, and performing feature normalization on the SIFT feature maps of the reference image and the image to be detected; Step 304: calculate the feature similarity of each feature point in the two normalized sift feature maps to obtain the similarity of each feature point in the two sift feature maps. ; Step 305: construct a pure black image with the same size as the image to be detected, filter out points in the SIFT feature map of the image to be detected whose feature similarity is less than a set threshold, determine the coordinates of all the filtered points, and set the pixel values ​​at the corresponding coordinate positions in the pure black image to white to obtain a black and white binary image; In step 306, an erosion and dilation operation is performed on the obtained black and white binary image to connect the white pixels into one or more regions, thereby preliminarily detecting the coordinates of the foreign body region in the image to be detected, and then the image to be detected is segmented based on the coordinates of the foreign body region to obtain a preliminarily detected foreign body sub-image.

7. The image-based foreign body and target anomaly detection method according to claim 1, 2, 3 or 4, characterized in that: In step 4, the brightness consistency processing process is as follows: For a set of authenticity identification image pairs, convert the initial detection foreign body sub-image and the reference sub-image, or the target suspected abnormal sub-image and the target reference sub-image into grayscale images, and obtain the grayscale mean of the grayscale image corresponding to the initial detection foreign body sub-image or the target suspected abnormal sub-image and normalized brightness histogram , s and the mean of the grayscale image corresponding to the benchmark sub-image and normalized brightness histogram ; Calculate the difference distance based on the normalized brightness histogram ,for ; Setting thresholds 、 ,and ; when When: First calculate the normalized histogram of the initial detection foreign body sub-graph or the target suspected abnormal sub-graph and its benchmark sub-graph 、 , and calculate the histogram 、 Cumulative distribution function of 、 , thus obtaining the mapping function Finally, the brightness consistency processing is performed based on the initial inspection of the foreign body sub-image or the target suspected abnormal sub-image according to the mapping function, that is, ,in, This is the initial inspection of foreign body sub-images or target suspected abnormal sub-images before brightness consistency processing. This is the initial inspection foreign body sub-image or target suspected abnormal sub-image after brightness consistency processing. is the pixel coordinate; when When , first according to the mean of the grayscale image pair 、 Calculate brightness consistency processing coefficient 、 ,for 、 , and then process the coefficient according to the brightness consistency 、 Perform brightness consistency processing on the initially detected foreign body sub-image or the target suspected abnormal sub-image, namely: .

8. The image-based foreign body and target anomaly detection method according to claim 1, 2, 3 or 4, characterized in that: In step 5, the cascade authenticity identification is performed on the authenticity identification image pair as follows: Extract the brightness difference features between the reference sub-image and the initial foreign body sub-image in the authenticity identification image, or the target suspected abnormal sub-image and the target reference sub-image based on the grayscale image , contrast difference characteristics , target contour difference characteristics ; Based on the light intensity difference feature , contrast difference characteristics , target contour difference characteristics , calculate the similarity distance metric ; judge Is it lower than the first foreign body threshold or the first target abnormality threshold: If so, the foreign body in the current preliminary foreign body sub-image is determined to be a real foreign body or the current target suspected abnormality sub-image is determined to be a target abnormality; otherwise, the foreign body in the current preliminary foreign body sub-image is determined to be a false foreign body or the current target suspected abnormality sub-image is determined to be a target suspected abnormality; A feature classification extractor is constructed based on a neural network. The trained feature classification extractor is used to extract features from the authenticity identification image pairs that are judged to be pseudo-foreign objects or suspected abnormal targets. The extracted features are then evaluated for similarity to obtain a similarity evaluation metric. Determine whether the similarity evaluation metric is lower than a second foreign body threshold or a second target abnormality threshold: If so, determine that the pseudo foreign body is a real foreign body or the target is abnormal.

9. An image-based foreign body and target anomaly detection system, characterized in that: The method according to any one of claims 1 to 8, wherein the detection system comprises: An image preprocessing unit, configured to obtain a reference image and an image to be detected, and perform noise reduction processing on the reference image and the image to be detected; an image alignment unit, configured to align the image to be detected with the reference image to obtain a mapping matrix between the image to be detected and the reference image; a foreign body segmentation unit, configured to perform feature extraction on the image to be detected and the reference image respectively, to obtain a feature map atlas of the image to be detected and a feature map atlas of the reference image, and to preliminarily detect the coordinates of the foreign body region in the image to be detected by using the difference between the feature points in the two feature maps, and then to segment the image to be detected based on the foreign body region coordinates to obtain a preliminarily detected foreign body sub-map; a target anomaly segmentation unit, configured to perform target detection on the reference image or directly anchor the target to obtain target region coordinates, and then segment the reference image based on the target region coordinates to obtain a target reference sub-image; a brightness processing unit, configured to map the coordinates of the foreign body region onto the reference image according to the mapping matrix, or map the coordinates of the target region onto the image to be detected according to the mapping matrix, and segment the reference image to obtain a reference sub-image or segment the image to be detected to obtain a target suspected abnormal sub-image based on the mapped coordinates, and then form an authenticity identification image pair with the reference sub-image and the corresponding foreign body sub-image of the initial inspection, or with the target suspected abnormal sub-image and the corresponding target reference sub-image, and perform brightness consistency processing; The cascaded authenticity identification unit is used to perform cascaded authenticity identification on the authenticity identification image pair to complete the detection of foreign matter or target anomaly in the image to be detected.

10. A terminal device, characterized in that: The terminal device is provided with: Memory, used to store programs; A processor is configured to execute the program stored in the memory, wherein when the program is executed, the processor is configured to execute the method according to any one of claims 1 to 8.

Images