Foreground extraction method, anomaly detection method, system thereof, and electronic device

Abstract

Embodiments of the present application provide a foreground extraction method, an anomaly detection method, a system thereof and an electronic device. The foreground extraction method comprises: obtaining a foreground outer expansion region near a foreground region in a to-be-extracted image according to position information of the foreground region in the to-be-extracted image; determining a template coarse positioning region in a template image according to the position information of the foreground region in the to-be-extracted image, the template coarse positioning region comprising a region corresponding to the foreground outer expansion region; performing feature matching on the foreground outer expansion region and the template coarse positioning region to obtain a first matching result; determining a template fine positioning region in the template image corresponding to the foreground region according to the matching result; and extracting the foreground according to at least a pixel value of each pixel in the foreground region and a pixel value of a corresponding pixel in the template fine positioning region. The foreground image obtained by the scheme has high accuracy, and various abnormal sample images of a detection model can be obtained using the foreground image, which can reduce the difficulty of collecting abnormal samples, save collection time and effort.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically to a foreground extraction method, an anomaly detection method, a foreground extraction system, an anomaly detection system, an electronic device, and a storage medium. Background Technology

[0002] In recent years, computer vision technology has been widely used in various fields, such as anomaly detection of objects under test.

[0003] In existing technologies, deep learning-based detection models are commonly used to detect anomalies in images of objects under test. While this method is highly efficient, it requires collecting and labeling a large number of anomaly samples as training images for the detection model. In actual production, the number of anomaly samples accounts for a very small proportion of the total sample size, typically less than 0.1%. Furthermore, manually labeling a large number of training images would incur significant labor costs. In short, obtaining a sufficient number of anomaly sample training images usually requires a considerable amount of time and effort. Summary of the Invention

[0004] This application is made in consideration of the above-mentioned problems. According to one aspect of this application, a foreground extraction method is provided, the method comprising: obtaining a foreground extension region located near the foreground region in the image to be extracted based on the location information of the foreground region in the image to be extracted; determining a template coarse localization region in a template image based on the location information of the foreground region in the image to be extracted, wherein the template coarse localization region includes a region corresponding to the foreground extension region; performing feature matching between the foreground extension region and the template coarse localization region to obtain a first matching result; determining a template fine localization region in the template image corresponding to the foreground region based on the first matching result; and extracting the foreground at least based on the pixel value of each pixel in the foreground region and the pixel value of the corresponding pixel in the template fine localization region.

[0005] For example, extracting the foreground based at least on the pixel value of each pixel in the foreground region and the pixel value of the corresponding pixel in the template fine positioning region includes: adjusting the brightness of the template fine positioning region or the foreground region so that the template fine positioning region and the foreground region are at the same brightness level after adjustment; and extracting the foreground by comparing the pixel value of each pixel in the foreground region at the same brightness level with the pixel value of the corresponding pixel in the template fine positioning region.

[0006] For example, adjusting the brightness of the template fine positioning area or the foreground area includes: determining a brightness mapping relationship based on the first matching result of the foreground expansion area and the template coarse positioning area; and adjusting the brightness of the template fine positioning area or the foreground area using the brightness mapping relationship.

[0007] Exemplarily, the foreground expansion area includes an annular area surrounding the foreground area, and the outer edge size of the template rough positioning area is larger than the outer edge size of the foreground expansion area.

[0008] Exemplarily, the width of the outer edge of the foreground expansion area is equal to n1 times the width of the foreground area, and the height of the outer edge of the foreground expansion area is equal to m1 times the height of the foreground area; the width of the outer edge of the template rough positioning area is equal to n2 times the width of the foreground area, and the height of the outer edge of the template rough positioning area is equal to m2 times the height of the foreground area, where n1 < n2 and / or m1 < m2.

[0009] Exemplarily, the method further includes: performing noise filtering on the foreground to obtain the filtered foreground.

[0010] According to the second aspect of the present application, an anomaly detection method is provided, including: inputting a to-be-detected image into a trained detection model to obtain an abnormal foreground in the to-be-detected image, where the detection model is trained by labeled training sample images, and the labeled training sample images are composite images of normal sample images and abnormal foregrounds extracted by the above foreground extraction method.

[0011] Exemplarily, the method further includes: constructing an abnormal foreground dataset based on the abnormal foreground; randomly selecting the current abnormal foreground from the abnormal foreground dataset; performing first brightness adjustment on the current abnormal foreground with a first probability and synthesizing it with the corresponding normal sample image to obtain a positive sample image that is greater than or equal to a first brightness threshold and includes an abnormal label; performing second brightness adjustment on the current abnormal foreground with a second probability and synthesizing it with the corresponding normal sample image to obtain a negative sample image that is less than a second brightness threshold and includes a normal label; where the labeled training sample images include positive sample images and negative sample images, the first probability is greater than the second probability, the second brightness threshold = the first brightness threshold * a preset ratio, and the preset ratio is less than 0.5.

[0012] Exemplarily, after obtaining the abnormal foreground in the to-be-detected image, the method further includes: determining a normal template fine positioning area corresponding to the abnormal foreground in the normal template image at least according to the position of the abnormal foreground in the to-be-detected image; respectively performing gray histogram statistical analysis on the abnormal foreground and the normal template fine positioning area, and comparing the statistical analysis results to determine the abnormal index value of the abnormal foreground.

[0013] For example, determining the normal template fine localization region corresponding to the abnormal foreground in the normal template image based at least on the position of the abnormal foreground in the image to be detected includes: obtaining an abnormal outward expansion region located around the abnormal foreground in the image to be detected based on the position information of the abnormal foreground in the image to be detected; determining the normal template coarse localization region in the normal template image based on the position information of the abnormal region in the image to be detected, wherein the normal template coarse localization region includes the region corresponding to the abnormal outward expansion region; performing feature matching on the abnormal outward expansion region and the normal template coarse localization region to obtain a second matching result; and determining the normal template fine localization region corresponding to the abnormal foreground in the normal template image based on the second matching result.

[0014] For example, before performing grayscale histogram statistical analysis on the abnormal foreground and the normal template fine-positioning area respectively, the method further includes: adjusting the brightness of the normal template fine-positioning area or the abnormal foreground so that the template fine-positioning area and the abnormal foreground are at the same brightness level after adjustment.

[0015] For example, grayscale histogram statistical analysis is performed on the abnormal foreground and the precise positioning area of ​​the normal template, and the statistical analysis results are compared to determine the abnormal index value of the abnormal area. This includes: obtaining the first grayscale histogram of the abnormal foreground and the second grayscale histogram of the precise positioning area of ​​the normal template, respectively; determining multiple abnormal index values ​​of the abnormal foreground based on the statistical analysis of the first grayscale histogram and the second grayscale histogram; and determining the comprehensive abnormal index value of the abnormal foreground based on the multiple abnormal index values.

[0016] According to a third aspect of this application, a foreground extraction system is also provided, comprising: an acquisition module, configured to acquire a foreground extension region located near the foreground region in the image to be extracted based on the location information of the foreground region in the image to be extracted; a first determination module, configured to determine a template coarse positioning region in a template image based on the location information of the foreground region in the image to be extracted, wherein the template coarse positioning region includes a region corresponding to the foreground extension region; a matching module, configured to perform feature matching between the foreground extension region and the template coarse positioning region to obtain a first matching result; a second determination module, configured to determine a template fine positioning region in the template image corresponding to the foreground region based on the first matching result; and an extraction module, configured to extract the foreground based at least on the pixel value of each pixel in the foreground region and the pixel value of the corresponding pixel in the template fine positioning region.

[0017] According to a fourth aspect of this application, an anomaly detection system is also provided, comprising: a detection module for inputting an image to be detected into a trained detection model to obtain an abnormal foreground in the image to be detected, wherein the detection model is trained from labeled training sample images, and the labeled training sample images are a composite image of a normal sample image and an abnormal foreground extracted using the aforementioned foreground extraction method.

[0018] According to a fifth aspect of this application, an electronic device is also provided, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the foreground extraction method or the anomaly detection method described above.

[0019] According to a sixth aspect of this application, a storage medium is also provided, on which program instructions are stored, which are used to execute the foreground extraction method or the anomaly detection method described above when the program instructions are executed.

[0020] In the above-described scheme of this application, based on the positional information of the foreground region in the image to be extracted, feature matching is performed on the foreground extension region in the image to be extracted and the template coarse localization region in the template image. Based on the matching results, the template fine localization region in the template image is determined. Finally, the foreground image is extracted based on the pixel value of each pixel in the foreground region and the pixel value of the corresponding pixel in the template fine localization region. The foreground image extracted by this scheme has high accuracy and represents anomalous foregrounds in real-world applications. By synthesizing this foreground image with a normal image, various anomalous sample training images for the detection model can be obtained, thus reducing the difficulty of collecting anomalous samples and saving time and effort in collection and annotation. Furthermore, this foreground extraction method has a relatively low computational load, saving computational resources.

[0021] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0022] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 A schematic flowchart of a foreground extraction method according to an embodiment of this application is shown;

[0024] Figure 2a and Figure 2b Schematic diagrams of the image to be extracted and the template image according to one embodiment of this application are shown respectively;

[0025] Figure 2c Another embodiment according to this application is shown. Figure 2b A schematic diagram of the coarse positioning area of ​​the template in the template image;

[0026] Figure 3 A schematic diagram of a foreground extraction method according to another embodiment of this application is shown;

[0027] Figure 4 A schematic flowchart of an anomaly detection method according to an embodiment of this application is shown;

[0028] Figure 5 A schematic diagram of an anomaly detection method according to another embodiment of this application is shown;

[0029] Figure 6 A schematic block diagram of a foreground extraction system according to an embodiment of this application is shown;

[0030] Figure 7 A schematic block diagram of an anomaly detection system according to an embodiment of this application is shown; and

[0031] Figure 8 A schematic block diagram of an electronic device according to one embodiment of this application is shown. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0033] As mentioned earlier, since the number of abnormal images in actual production is relatively small, and the annotation of abnormal image samples requires time and effort, obtaining enough abnormal sample training images usually takes a long time and effort.

[0034] To at least partially address the aforementioned problems, according to a first aspect of the embodiments of this application, a foreground extraction method is provided. This foreground extraction method can be used to extract foreground features from an image, such as abnormal regions within the image. Training sample images for training a detection model can be generated based on this foreground feature, and the trained detection model can then perform anomaly detection on the image to be detected. Figure 1 A schematic flowchart of a foreground extraction method 100 according to an embodiment of this application is shown. Figure 1 As shown, the foreground extraction method 100 may include the following steps: S110, S130, S150, S170 and S190.

[0035] Step S110: Based on the location information of the foreground region in the image to be extracted, obtain the foreground extension region located near the foreground region in the image to be extracted. Step S130: Based on the location information of the foreground region in the image to be extracted, determine the template coarse positioning region in the template image. The template coarse positioning region includes the area corresponding to the foreground extension region. It should be understood that the execution order of steps S110 and S130 can be arbitrary; they can be executed simultaneously (e.g., ...). Figure 1 The execution scheme shown can also be executed in any order, and this application does not restrict it.

[0036] According to embodiments of this application, the image to be extracted and the template image can be images of the same or the same target object. The target object can be any suitable object. For example, it can be industrial products of various types, forms, and uses. In one example, the target object can be an industrial product to be detected for anomalies, such as a workpiece of a certain model to be inspected. In this example, the image to be extracted can be an image of an abnormal workpiece containing anomalies, and the template image can be a standard reference image of that workpiece, such as a normal workpiece image. Of course, the image to be extracted and the template image can also be used for other purposes, including but not limited to product authenticity detection or foreground extraction in scenarios other than abnormal foreground extraction. For simplicity, the foreground extraction method 100 according to embodiments of this application will be described below using an abnormal image of the target object to be detected as the image to be extracted and a normal image of the target object to be detected as the template image.

[0037] Both the image to be extracted and the template image can be any suitable image, including two-dimensional black and white images and two-dimensional color images. They can be images of any suitable size and resolution, or images that meet preset requirements. For example, the image to be extracted and the template image can be color RGB images that meet the resolution requirements. Any existing or future image acquisition method can be used to acquire the image to be extracted and the template image. The image to be extracted and the template image can be the original image directly acquired by the image acquisition device, or an image after preprocessing. This preprocessing operation can include all operations to improve the visual effect of the image, increase image clarity, or highlight certain features in the image to facilitate foreground extraction. For example, preprocessing operations can include noise reduction operations such as filtering, or adjustments to image parameters such as image grayscale, contrast, and brightness.

[0038] It's easy to understand that regardless of the type of image (the image to be extracted) or the template image, their dimensions, colors, types, and other attributes can be identical. Furthermore, since both images target the same object, they can share common image features. For example, they could be images with the same parameters, captured using similar cameras, in similar shooting environments, and from similar shooting positions.

[0039] The foreground region can be any image area in the image to be extracted that contains the desired foreground element. In the example where the image to be extracted is an anomalous sample image, the foreground region can be an area where the target object being detected exhibits anomalies. For example, the area containing an abnormal part of the workpiece to be detected, such as an incorrectly installed part, a damaged part, or a contaminated part. This foreground region can be a region predetermined using any suitable method. For example, it can be an image area containing an abnormal foreground element in the image to be extracted, obtained through manual annotation or machine annotation.

[0040] The foreground region can be any shape and size. For example, and not limitingly, in the image to be extracted, the foreground region can be the area containing a pre-labeled rectangular bounding box containing the abnormal foreground. The positional information of the foreground region in the image to be extracted can be represented, for example, by the coordinates of the rectangular bounding box, such as the coordinates of the top-left corner of the bounding box and the width and height of the bounding box. Of course, the foreground region can also be a region of other shapes, and its positional information in the image to be extracted can be represented by other suitable positional information.

[0041] In step S110, based on the determined location information of the foreground region in the image to be extracted, a foreground extension region located near the foreground region can be obtained in the image to be extracted. The foreground extension region can be an extended area obtained by expanding the foreground region in any direction. According to an embodiment of this application, the foreground extension region can be an image region adjacent to the foreground region. Optionally, the foreground extension region can be a left-side extension region obtained by expanding the left edge of the foreground region by a first dimension. Alternatively, the foreground extension region can be a left-side extension region obtained by expanding the left edge of the foreground region by a first dimension to the left and a right-side extension region obtained by expanding the right edge of the foreground region by a second dimension to the right. Of course, the foreground extension region can also be a foreground extension region obtained by expanding in other directions along the edge of the foreground region. The foreground extension region can have any suitable shape and any suitable size, as long as it is on the image to be extracted and located near the foreground region.

[0042] Figure 2a and Figure 2b Schematic diagrams of the image to be extracted and the template image according to one embodiment of this application are shown respectively. Figure 2aAs shown, the foreground region 210 in the image to be extracted can be determined first (shown in the area within the black solid line box in the figure). The foreground region 210 is, for example, a rectangular region, and its position coordinates in the image to be extracted are, for example, (x0, y0, w, h), where x0 and y0 represent the horizontal and vertical coordinates of the center point of the rectangular region, respectively, and w and h represent the width and height of the rectangular region, respectively. The outward expansion dimension is, for example, an equal-distance expansion dimension d along the four directions of the foreground region, thus obtaining the foreground expansion region 220 (shown in the area filled with diagonal lines in the figure). The position of its outer edge is, for example, (x0, y0, w+d, h+d). Here, the foreground expansion region 220 is a square ring region with a width of d.

[0043] According to embodiments of this application, any suitable method can be used to obtain the foreground expansion region. Exemplarily, but not limitingly, the foreground region can be a free polygonal region. The outer contour of the polygonal region containing the foreground region can be determined based on the obtained position information of the foreground region in the image to be extracted. Then, any suitable equidistant expansion method can be used to obtain the position information of the outer contour of the expanded foreground region. Alternatively, if the foreground region is a regular polygonal region such as a rectangle, the vertex position coordinates of the foreground expansion region can be calculated based on the center coordinates of the foreground region in the image to be extracted, the width and height of the region, and preset expansion width and preset expansion height. Other suitable methods can also be used to determine the position of the foreground expansion region.

[0044] As mentioned above, the image to be extracted and the template image can be images with common image features. Therefore, in step S130, a coarse localization region of the template image can be determined based on the position information of the foreground region in the image to be extracted. Exemplarily, but not limitingly, the image to be extracted and the template image are the same size, and the positions of various parts of the target object in the images are also approximately the same. Therefore, based on the position information of the foreground region in the image to be extracted, a region in the template image that is at the same position as the foreground region can be determined accordingly. Exemplarily, but not limitingly, this position information may include the position center information and region size information of the foreground region. (See reference...) Figure 2a The foreground area can be a rectangular region, with its center (shown as a black dot in the diagram) at coordinates (x0, y0), a width of w, and a height of h. Then... Figure 2bThe corresponding center 230 (shown as a black dot in the figure) is located in the template image. Then, a coarse template positioning area can be determined in the template image using a method similar to that used in step S110 to obtain the foreground expansion area. This coarse template positioning area can be an area of ​​any shape, including the region corresponding to the foreground expansion area. For the foreground expansion area, a corresponding region in the template image with the same positional information can be determined, i.e., its corresponding region. The coarse template positioning area can be equal to or larger than this corresponding region. For example, if the foreground area is expanded outward along a first direction by a first distance in the image to be extracted to obtain the foreground expansion area, then in the template image, the coarse template positioning area can be obtained by expanding the region corresponding to the foreground area outward along the first direction by a second distance, and the second distance is not less than the first distance.

[0045] In a specific example, such as Figure 2b As shown, the coarse positioning area of ​​the template in the template image, determined based on the position information of the foreground area, can be a large rectangular area 240 (the area filled with diagonal lines in the figure) whose size is larger than the area corresponding to the foreground expansion area. In this example, a large rectangular area 240 with a position center of (x0, y0), a width of 4w, and a height of 4h can be located in the template image based on the position center (x0, y0) of the foreground area and the width w and height h of the foreground area. Figure 2c Another embodiment according to this application is shown. Figure 2b This is a schematic diagram of the coarse positioning region of the template in the template image. In another specific example, such as... Figure 2c As shown, the template coarse positioning area in the template image determined based on the position information of the foreground area can also be a square ring area 240' (the area filled with diagonal lines in the figure) whose area size is larger than the area corresponding to the foreground expansion area. The inner edge of the square ring area 240' can be determined based on the position information of the foreground area.

[0046] Step S150: Perform feature matching between the foreground expansion area and the template coarse positioning area to obtain the first matching result.

[0047] As mentioned earlier, since the image to be extracted and the template image are images of the same target object, they share the same image features. Furthermore, the template coarse localization region includes the region corresponding to the foreground extension region. Therefore, the template coarse localization region and the foreground extension region necessarily have the same image features. In step S150, feature matching can be performed on these two regions to obtain the feature matching result.

[0048] Step S150 can be implemented using any existing or future feature matching method. In one example, step S150 may further include steps S151 and S152. In step S151, any suitable feature point extraction method can be used to extract the first feature points in the foreground extension region of the image to be extracted and the second feature points in the template coarse localization region of the template image. Then, in step S152, feature matching can be performed on the obtained first and second feature points to obtain a first matching result. This step can be implemented using any suitable matching algorithm. Optionally, the Random Sample Consensus (RANSAC) algorithm can be used to perform feature matching on the two regions to obtain the optimal homography matrix of the matching model. Alternatively, the Scale Invariant Feature Transform (SIFT) algorithm can be used to match the feature points of the two regions. In other examples, other suitable image feature matching algorithms can also be used to match the image features of the two regions.

[0049] Step S170: Based on the first matching result, determine the template fine-positioning area in the template image that corresponds to the foreground area.

[0050] After obtaining the feature matching results between the coarse localization region of the template image and the foreground expansion region of the image to be extracted in step S150 using the feature point matching algorithm, the region corresponding to the foreground region in the template image can be accurately located based on these feature matching results. For example, in the above steps, the RANSAC algorithm can be used to obtain the feature point matching relationship between the foreground expansion region and the coarse localization region of the template. That is, the first feature points a1, a2, a3...a1 in the foreground expansion region can be obtained. n The second feature points b1, b2, b3...b in the coarse localization region of the template n The correspondence between them. Then, in step S170, based on the pixel position of each feature point and the correspondence between feature points, the template image is aligned to the image to be extracted.

[0051] Alignment between the template image and the image to be extracted can be achieved using any suitable method. For example, an affine transformation can be performed on the template image based on the matching results of feature points. It's easy to understand that after the affine transformation, the pixel positions of the first feature point in the foreground expansion region and the pixel positions of the second feature point in the template coarse localization region can remain consistent. Furthermore, based on the positional information of the foreground region in the image to be extracted, the fine localization region of the template corresponding to the foreground region can be precisely located in the affine-transformed template image. Since the foreground expansion region is located near the foreground region, after performing an affine transformation on the template image based on the feature point matching results of the foreground expansion region and the template coarse localization region, the fine localization region of the template image can be precisely aligned with the foreground region. Thus, pixel-level precise alignment of the feature-matched foreground region and the fine localization region of the template can be achieved in the same coordinate system.

[0052] Step S190: Extract the foreground based on at least the pixel value of each pixel in the foreground area and the pixel value of the corresponding pixel in the template fine positioning area.

[0053] Step S170 described above enables precise alignment of the foreground area with the template's fine-positioning area. After precise alignment, the foreground can be extracted by comparing the pixel values ​​of each pixel in both areas. Any suitable method can be used to compare the pixel values ​​of each pixel in both areas. Optionally, the difference in pixel values ​​of pixels at the same position in both areas can be directly compared, and the foreground image can be extracted based on this difference. Alternatively, an indirect comparison method can be used, for example, by first transforming the pixel values ​​of each pixel in either area to an intermediate value, and then extracting the foreground image by comparing the differences between the transformed pixel values.

[0054] Foreground can be extracted using any suitable foreground image extraction method. Exemplarily, and not limitingly, the image size of the extracted foreground image can be equal to the size of the foreground region. According to embodiments of this application, in the extracted foreground image, the pixel value of each pixel can represent the difference between the pixel value of the corresponding pixel in the foreground region of the image to be extracted and the pixel value of the corresponding pixel in the template fine-positioning region of the template image. Here, since there is foreground, such as an abnormal foreground, in the foreground region, for simplicity, each pixel of the abnormal foreground can be called a foreground pixel. Since the template fine-positioning region does not contain this abnormal foreground, each pixel of the template fine-positioning region can be called a background pixel. It is easy to understand that the pixel values ​​of the foreground pixels in the foreground region and the corresponding background pixels in the template fine-positioning region are different, and the difference between their pixel values ​​is not 0 and is within the pixel range (0, 255). Since the pixel values ​​of the background pixels in the foreground region and the corresponding background pixels in the template fine-positioning region are the same, that is, the difference between their pixel values ​​is 0, the pixel values ​​of the background pixels in the finally extracted foreground image are all 0. Therefore, a foreground image with 0 background pixels and the same size as the foreground region can be obtained.

[0055] The foreground extraction method of this application can be applied to the extraction of abnormal foreground from abnormal samples. The pixel values ​​of the obtained abnormal foreground region pixels can represent the difference between the pixel values ​​of abnormal pixels and normal pixels. On the one hand, the foreground image can accurately represent the location of the anomaly and the pixel value of the pixel containing the anomaly, thus the accuracy of the abnormal foreground represented by the image is higher. On the other hand, based on this, it is also convenient to perform various basic transformations on the abnormal image to generate more abnormal sample images that can simulate real anomalies. This can solve the problem of the scarcity and difficulty in collecting abnormal sample images in the prior art.

[0056] According to the above scheme, based on the positional information of the foreground region in the image to be extracted, feature matching is performed on the foreground extension region in the image to be extracted and the template coarse localization region in the template image. Based on the matching results, the template fine localization region in the template image is determined. Finally, the foreground image is extracted by comparing the pixel value of each pixel in the foreground region with the corresponding pixel value in the template fine localization region. The foreground image extracted by this scheme has high accuracy and represents anomalous foregrounds in real-world applications. By synthesizing this foreground image with a normal image, various anomalous sample training images for the detection model can be obtained, thus reducing the difficulty of collecting anomalous samples and saving time and effort in collection and annotation. Furthermore, this foreground extraction method has a relatively low computational cost, saving computational resources.

[0057] Exemplarily, the foreground expansion area includes an annular area surrounding the foreground area, and the outer edge size of the template rough positioning area is larger than the outer edge size of the foreground expansion area. According to an embodiment of the present application, the foreground expansion area may also be an annular area adjacent to the periphery of the foreground area. That is, the foreground expansion area may be an expansion area that expands in all directions along the four sides with the center position of the foreground area as the center. The inner edge of the foreground expansion area may be the outer edge of the foreground area, and the outer edge of the foreground area may be the expanded edge of the outer edge of the foreground area. For example, if the foreground area is a square area with side length a, the outer edge of the foreground expansion area may be a large square with side length greater than a, or the outer edge of the foreground expansion area may also be other shapes outside the square area. In short, the outer edge of the foreground expansion area is outside the outer edge of the foreground area. Optionally, the foreground expansion area may be a regular annular area, for example, it may be an annular area that uniformly expands a certain size along the outer edge of the foreground area. Alternatively, the foreground expansion area may also be an irregular annular area, such as an annular area that randomly expands different sizes in all directions along the outer edge of the foreground area. The same method as step S110 can be used to expand the foreground area to obtain the foreground expansion area surrounding the foreground area, which is easily understood by those skilled in the art and will not be elaborated here.

[0058] According to the above solution, the foreground expansion area can be obtained by expanding in all directions along the periphery of the foreground area. Since the foreground expansion area covers the surrounding area of the foreground area, feature points covering various positions around the foreground area can be matched in the feature matching between the foreground expansion area and the template rough positioning area, and thus the accuracy of positioning the template fine positioning area can be significantly improved. Thereby, the accuracy of foreground extraction can be improved.

[0059] Of course, in practical applications, in some special scenarios, the foreground expansion area can also be obtained by expanding in any one or more directions along the periphery of the foreground area.

[0060] Exemplarily, the width of the outer edge of the foreground expansion area is equal to n1 times the width of the foreground area, and the height of the outer edge of the foreground expansion area is equal to m1 times the height of the foreground area; the width of the outer edge of the template rough positioning area is equal to n2 times the width of the foreground area, and the height of the outer edge of the template rough positioning area is equal to m2 times the height of the foreground area, where n1 < n2 and / or m1 < m2.

[0061] According to an embodiment of this application, the foreground area can be expanded relatively evenly with the center of the foreground area as the center to obtain an extended foreground area. The size of the outer edge of the extended foreground area can be a multiple of the size of the foreground area. The size of the outer edge of the template coarse positioning area can also be a multiple of the size of the foreground area. Furthermore, the size of the template coarse positioning area can be larger than the size of the extended foreground area. Exemplarily and not limitingly, the foreground area can be a first rectangular area with a width and height of a*b centered at point O, the outer edge of the extended foreground area can be a second rectangular area with a larger size centered at point O, and the outer edge of the template coarse positioning area can also be a third rectangular area with a larger size centered at point O. For example, the second rectangle has a width of 2a and a height of 3b, while the third rectangle can have a width of 3a and a height of 4b. Various possible implementations of this solution are readily understood by those skilled in the art and will not be elaborated here.

[0062] The foreground expansion area and the template coarse positioning area can be determined using any suitable method. Optionally, the position O of the center point of the foreground area in the image to be extracted, as well as the width a and height b of the foreground area, can be determined first. Then, the center position O1 of the template area can be located based on these coordinates. A second rectangle with a width of 2a and a height of 3b, centered at O, can be defined in the image to be extracted as the outer edge of the foreground expansion area, and this outer edge can be used as the inner edge of the foreground expansion area. A third rectangle with a width of 3a and a height of 4b, centered at O1, can be defined in the template image as the outer edge of the template coarse positioning area. The entire area of ​​this third rectangle can be directly used as the template coarse positioning area, or the edge of the area in the template image corresponding to the first rectangle can be used as the inner edge of the template coarse positioning area. Of course, other suitable methods can also be used to determine the foreground expansion area and the template coarse positioning area.

[0063] According to the above scheme, the outer edge dimensions of the foreground expansion region and the template coarse positioning region can be multiples of the foreground region. Furthermore, the outer edge dimension of the template coarse positioning region is larger than that of the foreground expansion region. This scheme can also cover more areas surrounding the foreground region, allowing for the extraction of more feature points between the foreground expansion region and the template coarse positioning region for feature matching, thereby improving the accuracy of the template fine positioning region and the precision of foreground extraction. Moreover, this scheme is simpler, requires less computation, and saves more computational resources.

[0064] For example, step S190 extracts the foreground based at least on the pixel value of each pixel in the foreground area and the pixel value of the corresponding pixel in the template fine positioning area, including steps S191 and S192.

[0065] In step S191, the brightness of the template fine positioning area or the foreground area is adjusted so that the template fine positioning area and the foreground area are at the same brightness level after adjustment.

[0066] Since the image to be extracted and the template image may be captured at different times, there may be slight differences in the shooting parameters such as exposure. Therefore, the brightness levels of the pixels in the two images may be inconsistent, especially the brightness levels of the foreground area and the template fine-positioning area. Figure 2a The image to be extracted shows a higher brightness in the foreground area, while... Figure 2b The brightness level of the area corresponding to the foreground region in the template image shown is relatively low. Therefore, there is a significant difference in brightness levels between the foreground area and the template fine-positioning area in this example. In this case, the brightness of either one can be adjusted first in step S191 to make their overall brightness levels comparable. Optionally, the brightness of each pixel in the foreground area of ​​the image to be extracted can be uniformly adjusted based on the brightness of each pixel in the template fine-positioning area. Alternatively, the brightness of each pixel in the template fine-positioning area can be adjusted based on the brightness of each pixel in the foreground area.

[0067] According to the above scheme, the brightness of pixels in the foreground region of the image to be extracted or the template fine-positioning region of the template image can be calibrated. Then, the difference in pixel values ​​between pixels in two regions with similar brightness levels can be compared, and the foreground can be extracted based on the comparison results. The foreground image obtained by this scheme has higher accuracy and can more intuitively reflect the distribution of abnormal details. Therefore, the abnormal foreground can be generalized to obtain more simulated abnormal samples, thereby facilitating the collection of abnormal samples.

[0068] For example, step S191 adjusts the brightness of the template fine positioning area or the foreground area, including steps S191.1 and S191.2.

[0069] In step S191.1, the brightness mapping relationship is determined based on the first matching result between the foreground expansion region and the template coarse positioning region. According to embodiments of this application, the feature point matching result between the foreground expansion region and the template coarse positioning region can be obtained, for example, based on RANSAC matching. For instance, the positional correspondence between the first and second feature points can be obtained. Then, based on the brightness values ​​of each pixel in the image to be extracted and the template image corresponding to the first and second feature points respectively, the brightness mapping relationship of the pixels at the feature point positions in the image to be extracted and the template image can be determined. Any existing or future researched method can be used to determine the above brightness mapping relationship; for example, the least squares method can be used to obtain the brightness mapping formula. Alternatively, a trained model can be used to automatically fit the above brightness mapping relationship.

[0070] In step S191.2, the brightness of the template fine-positioning area or the foreground area is adjusted using the brightness mapping relationship. After determining the brightness mapping relationship of pixels at corresponding feature point positions in the image to be extracted and the template image, the brightness of the foreground area or the template fine-positioning area can be calibrated based on this brightness mapping relationship. For example, the mapping relationship between the pixel brightness of the second feature point on the template image and the pixel brightness of the first feature point on the image to be extracted can be obtained using the least squares method. Then, this brightness mapping relationship can be applied to each pixel in the template fine-positioning area to obtain the brightness value of each pixel in the calibrated template fine-positioning area. Alternatively, the mapping relationship between the pixel brightness of the first feature point on the image to be extracted and the pixel brightness of the second feature point on the template image can also be obtained, and the brightness of each pixel in the foreground area can be calibrated based on this mapping relationship to obtain the calibrated foreground area.

[0071] According to the above scheme, the brightness mapping relationship can be determined based on the matching results of feature points between the foreground expansion area and the template coarse positioning area, and the foreground area or template fine positioning area can be calibrated based on this brightness mapping relationship. This brightness calibration scheme is more accurate, thus obtaining a precisely calibrated template fine positioning area or foreground area, thereby improving the accuracy of the extracted foreground image. Furthermore, this scheme has a smaller computational load, saving computational resources and also helping to improve the efficiency of foreground extraction.

[0072] For example, after step S190, the foreground extraction method 100 further includes step S195, which performs noise filtering on the foreground to obtain the filtered foreground.

[0073] To obtain a more accurate foreground image, denoising processing can be performed on the extracted foreground image after step S190. Any suitable image denoising method can be used to denoise the foreground image. For example, mean filtering, median filtering, minimum variance mean filtering, K-nearest neighbor smoothing filtering, symmetric nearest neighbor mean filtering, and Sigmund smoothing filtering can be used. This approach is readily understood by those skilled in the art and will not be elaborated upon further here.

[0074] The image filtering and morphological operations described above can significantly reduce noise interference in the foreground image, which helps to generate more universal and accurate abnormal sample images.

[0075] Figure 3A flowchart illustrating a foreground extraction method according to another embodiment of this application is shown. As shown, firstly, an image to be extracted and a template image for the same target object can be acquired. For example, an abnormal sample image and a normal template image for a certain type of workpiece can be acquired. Abnormal regions in the abnormal sample image can be determined by manual annotation. Based on the position information of the abnormal region in the abnormal sample image, the abnormal region can be expanded to obtain an abnormal expansion area. A coarse template positioning area can also be determined in the normal template image, which may include the region corresponding to the abnormal expansion area. Secondly, feature points can be extracted from the acquired abnormal expansion area and the coarse template positioning area, and RANSAC matching can be performed. Based on the matching result, an affine transformation can be performed on the normal template image to accurately align the normal template image and the image to be extracted. Based on this, a fine template positioning area is then determined in the normal template image. Next, based on the feature point matching result, the brightness mapping relationship of pixels at corresponding feature point positions in the normal template image and the image to be extracted can be determined, and the brightness of pixels in the fine template positioning area can be calibrated according to the brightness mapping relationship. Finally, the foreground image is extracted based on the difference between the pixel value of each pixel in the abnormal region and the pixel value of the corresponding pixel in the calibrated template fine-positioning region.

[0076] According to a second aspect of this application, an anomaly detection method 400 is provided. Figure 4 A schematic flowchart of an anomaly detection method 400 according to an embodiment of this application is shown. As shown, the anomaly detection method 400 includes step S450, inputting the image to be detected into a trained detection model to obtain the abnormal foreground in the image to be detected. The image to be detected can be an image of the target object to be detected. It can be any suitable image. The detection model can be any suitable network model, such as a trained anomaly detection model like the YOLOv5 network model. The detection model is trained using labeled training sample images, which are composite images of normal sample images and the abnormal foreground and normal images extracted using the foreground extraction method 100 described above. Any suitable image synthesis method can be used to synthesize the abnormal foreground and normal sample images to obtain the training sample image. For example, the training sample image can include a foreground region and a background region. The foreground region can be an image region obtained by performing basic transformations such as rotation, scaling, and affine transformation on the abnormal foreground image extracted from the abnormal sample image using the foreground extraction method described above. The background region is the image region other than the normal sample image occluded by the foreground region. It's easy to understand that a single image with an anomalous foreground can be transformed to generate multiple training sample images containing the anomalous foreground, facilitating the collection of more anomalous sample images. The annotation information of the training sample images can include the location information of the anomalous foreground in the image and the anomalous category information.

[0077] It's easy to understand that by feeding a large number of labeled training sample images into an initial detection model for training, a trained detection model can be obtained. Inputting the image to be detected into the trained detection model yields the model's detection result. When the image to be detected contains anomaly regions, the trained detection model can accurately output anomaly information, such as the location of the anomaly region, i.e., the location of the abnormal foreground. For example, it can output a rectangular bounding box containing the abnormal foreground. Furthermore, based on this rectangular bounding box, an image of the abnormal foreground region corresponding to that bounding box can be obtained from the image to be detected.

[0078] According to the above scheme, the aforementioned foreground extraction method can be used to extract abnormal foregrounds from abnormal sample images to obtain training sample images for an abnormal foreground synthesis detection model. The synthesized training sample images can then be used to train the detection model. The trained detection model can then accurately detect abnormal foregrounds in the image to be detected. This scheme allows for the synthesis of more abnormal training sample images from a small number of abnormal sample images. This significantly reduces the time and difficulty of collecting abnormal training samples, enabling rapid training of the detection model and thus improving the detection efficiency of the target object. Furthermore, since the extracted abnormal foregrounds are from the real scene, the trained model is more consistent with the requirements and has higher detection accuracy.

[0079] For example, prior to step S450, the anomaly detection method 400 further includes steps S410, S420, and S430. According to embodiments of this application, the anomaly detection method 400 may also include the following step of collecting training sample images.

[0080] In step S410, an abnormal foreground dataset is constructed based on the abnormal foreground. For example, taking the anomaly detection of a certain type A workpiece as an example, images of abnormal workpieces generated during production over a certain period can be collected. The abnormal regions can be manually labeled in the images. Then, a fixed-size abnormal foreground images can be extracted using the foreground extraction method 100, and each abnormal foreground image can be stored in the abnormal foreground dataset. For example, the abnormal foreground data can be created as a dataset file in the format of an abnormal shape .npz.

[0081] In step S420, a current anomalous foreground is randomly selected from the anomalous foreground dataset. The current anomalous foreground used to train the detection model can be randomly selected from the anomalous foreground dataset. One or more anomalous foregrounds can be randomly selected as the current anomalous foreground.

[0082] In step S430, the current abnormal foreground can be adjusted in brightness with a first probability and synthesized with the corresponding normal sample image to obtain a positive sample image that is greater than or equal to a first brightness threshold and includes an abnormal label. The current abnormal foreground can be adjusted in brightness with a second probability and synthesized with the corresponding normal sample image to obtain a negative sample image that is less than a second brightness threshold and includes a normal label. The labeled training sample images include positive and negative sample images, with the first probability greater than the second probability. The second brightness threshold = the first brightness threshold * a preset ratio, where the preset ratio is less than 0.5. Both the first and second probabilities can be any suitable probability value, and the sum of the first and second probabilities is 100%. The first and second brightness thresholds can also be any suitable brightness thresholds. Furthermore, different first and second brightness thresholds can be set for different target objects or corresponding to different detection requirements. In one example, the first probability is, for example, 80%, and the second probability is, for example, 20%. That is, the probability of generating a positive sample can be 80%, and the probability of generating a negative sample can be 20%. In the example where the training sample image is a grayscale image, the brightness threshold can be represented as a grayscale threshold. The first brightness threshold can be represented by a first grayscale threshold. The second brightness threshold can be represented by a second grayscale threshold. The first grayscale threshold is, for example, 100, and the second grayscale threshold is, for example, less than 50, such as 20. That is, in step S430, a positive sample image including an anomaly label can be synthesized with an 80% probability, and a negative sample image including a normal label can be synthesized with a 20% probability. The positive sample image can be obtained by synthesizing the current abnormal foreground with a normal sample image after overall brightness adjustment. For example, the average brightness value or average grayscale value of the abnormal foreground in the positive sample image can be greater than or equal to 100. The average brightness value or average grayscale value of the abnormal foreground in the negative sample image can be less than 20.

[0083] For example, before synthesizing the anomalous foreground and normal sample images, a random affine transformation can be performed on the anomalous foreground to obtain one or more transformed anomalous foregrounds. The brightness of this transformed anomalous foreground can then be adjusted to generate more positive and negative sample images. Labels can then be regenerated online based on the acquired positive and negative sample images to obtain labeled positive and negative sample images. Compared to the manual labeling scheme in existing technologies, this online labeling scheme optimizes the data reading module of the detection model. Furthermore, performing a random affine transformation on the anomalous foreground in each training round to generate new label data can greatly expand the diversity of the original anomalous samples and the generalization ability of the detection model.

[0084] According to embodiments of this application, training sample images obtained using the above-described method for synthesizing training sample images can be used to perform multiple rounds of iterative training on a detection model such as YOLOv5. Furthermore, in each round of iterative training, the above-described method for synthesizing training sample images and the current training sample image used for each round of training can be employed. This allows for the synthesis of a larger number of accurate anomaly training sample images from a smaller number of anomaly sample images to train the detection model. Therefore, the training efficiency of the detection model can be significantly improved, as can the anomaly detection efficiency of the detection model. Moreover, since the synthesized training sample images also include a relatively small number of negative sample training images, the over-detection rate of the detection model can be reduced to some extent.

[0085] For example, after step S450, the anomaly detection method 400 further includes steps S460 and S470. According to embodiments of this application, after obtaining the abnormal foreground in the image to be detected through the detection model, the abnormal foreground can also be post-processed to determine the final detection result.

[0086] In step S460, at least based on the position of the aberrant foreground in the image to be detected, a precise localization region of the normal template corresponding to the aberrant foreground in the normal template image is determined. It is readily understood that this precise localization region of the normal template can be a region that precisely corresponds to the position and features of the aberrant foreground. Any suitable image processing method can be used to determine the precise localization region of the normal template.

[0087] In step S470, gray-level histogram statistical analysis is performed on the abnormal foreground and the finely localized area of ​​the normal template, respectively, and the statistical analysis results are compared to determine the abnormality index value of the abnormal foreground. For example, the distribution probability of different gray levels in the abnormal foreground area of ​​the image to be detected and the distribution probability of different gray levels in the finely localized area of ​​the normal template in the normal template image can be statistically analyzed. One or more quantifiable statistical indicators of the gray levels of each pixel in their respective regions can also be further analyzed based on their respective gray-level histograms. The differences between the statistical analysis results of the gray-level histograms of the two can be compared to determine the abnormality index value of the abnormal foreground. The abnormality index value can be any quantifiable value that can reflect the difference in gray levels between the pixels of the abnormal foreground and the coarsely localized area of ​​the normal template; this application does not limit it. For example, the abnormality index value can be an indicator that reflects the similarity of pixels in the two regions. Alternatively, it can be an indicator that reflects the degree of deviation between pixels in the two regions. Or, it can be a comprehensive index value of multiple abnormality index values.

[0088] According to the above scheme, the precise localization region of the normal template corresponding to the abnormal foreground in the acquired image to be detected can be determined. Then, by performing histogram statistical analysis on the abnormal foreground and the precise localization region of the normal template, the quantifiable anomaly index value of the abnormal foreground can be determined based on the difference between the two analysis results. This scheme not only detects abnormal regions through the model but also further determines the anomaly index value of the abnormal regions through post-processing. Thus, the accuracy of anomaly detection can be significantly improved through post-processing re-judgment.

[0089] For example, step S460, determining the precise localization region of the normal template in the normal template image corresponding to the abnormal foreground based at least on the position of the abnormal foreground in the image to be detected, includes:

[0090] In step S461, based on the location information of the abnormal foreground in the image to be detected, the abnormal outward expansion region located around the abnormal foreground is obtained in the image to be detected. This step can be implemented using a method similar to that described in step S110 above, and will not be repeated here.

[0091] In step S462, based on the location information of the abnormal region in the image to be detected, a normal template coarse localization region in the normal template image is determined, wherein the normal template coarse localization region includes the region corresponding to the abnormal outward expansion region. This step can be implemented using a method similar to that described in step S130 above, and will not be repeated here.

[0092] In step S463, feature matching is performed on the abnormal expansion region and the normal template coarse localization region to obtain a second matching result. This step can be implemented using a method similar to that described in step S150 above, and will not be repeated here.

[0093] In step S464, based on the second matching result, the precise localization region of the normal template corresponding to the abnormal foreground in the normal template image is determined. This step can be implemented using a method similar to that described in step S170 above, and will not be repeated here.

[0094] According to the above scheme, after the abnormal foreground in the image to be detected is output by the detection model, the abnormal expansion area and the normal template coarse localization area in the normal template image can be determined based on the position of the abnormal foreground in the image to be detected. Then, based on feature matching of the abnormal expansion area and the normal template coarse localization area, the normal template fine localization area corresponding to the abnormal foreground in the normal template image can be accurately located. Furthermore, the abnormality index value of the abnormal foreground can be determined by statistical analysis of the gray-level histograms of the abnormal foreground and the normal template fine localization area. This scheme can obtain more accurate abnormality index values ​​for the abnormal foreground, thereby improving the detection accuracy of anomaly detection.

[0095] For example, before performing grayscale histogram statistical analysis on the abnormal foreground and the normal template fine-positioning area in step S470, the anomaly detection method 400 further includes step S465, adjusting the brightness of the normal template fine-positioning area or the abnormal foreground so that the adjusted template fine-positioning area and the abnormal foreground are at the same brightness level. Step S465 can use the same method as steps S191.1 and S191.2 in the aforementioned foreground extraction method 100 to calibrate the brightness of the normal template fine-positioning area or the abnormal foreground. Those skilled in the art can understand the implementation method of this step by reading the foregoing scheme, and it will not be described in detail here.

[0096] According to the above scheme, the brightness of the precise positioning area of ​​the normal template or the abnormal foreground is adjusted to unify them to the same brightness level before grayscale histogram statistical analysis is performed. This scheme yields more accurate analysis results and provides more precise and intuitive quantitative indicators that reflect the degree of abnormality of the abnormal foreground.

[0097] For example, step S470 performs grayscale histogram statistical analysis on the abnormal foreground and the normal template fine-positioning area respectively, and compares the statistical analysis results to determine the abnormal index value of the abnormal area, including steps S471 and S472.

[0098] In step S471, the first gray-level histogram of the abnormal foreground and the second gray-level histogram of the normal template fine-positioning area are obtained respectively. In one example, after brightness calibration of the normal template fine-positioning area, the gray-level values ​​of each pixel in the brightness-calibrated normal template fine-positioning area and the abnormal foreground can be statistically analyzed to obtain the first gray-level histogram of the abnormal foreground and the second gray-level histogram of the normal template fine-positioning area. Any existing or future gray-level histogram acquisition method can be used to obtain the first and second gray-level histograms.

[0099] In step S472, multiple abnormal indicator values ​​for the abnormal foreground are determined based on statistical analysis of the first and second gray-level histograms, and a comprehensive abnormal indicator value for the abnormal foreground is determined based on these multiple abnormal indicator values. According to embodiments of this application, statistical analysis can be performed on the gray-level histograms of the abnormal foreground and the template precision positioning area to obtain various quantitative statistical indicators that can represent gray-level statistics. Then, the same quantitative statistical indicator of the two can be compared, and quantitative values ​​representing the differences between the two, such as differences or ratios of each quantitative indicator, can be obtained. The difference value of each quantitative indicator can be used as an abnormal indicator for the abnormal foreground, or the difference value after further statistical analysis of the differences of multiple quantitative indicators can be used as an abnormal indicator for the abnormal foreground. Finally, a comprehensive abnormal indicator value for the abnormal foreground can be obtained by combining multiple abnormal indicators.

[0100] As is easily understood, the multiple anomaly indicators and the comprehensive anomaly indicator identified in this step can all reflect the grayscale difference between the abnormal foreground and the finely positioned area of ​​the normal template, thus quantifying the degree of anomaly of the abnormal foreground. This helps users intuitively understand the degree of anomaly of the abnormal foreground.

[0101] By way of example, and not limitation, multiple outlier values ​​may include the difference between the means of the first gray-level histogram and the second gray-level histogram. The difference dX between the root mean squares of the first and second gray-level histograms rms The following are considered: the difference in skewness (dSkew) between the first and second gray-level histograms; the difference in waveform indices (dS) between the first and second gray-level histograms; the difference in peak indices (dC) between the first and second gray-level histograms; and the difference in the AD test statistic (AD) between the integral curves of the first and second gray-level histograms. Based on these multiple outlier values, a comprehensive outlier index value for the anomalous foreground is determined, including calculating the comprehensive outlier index value A using the following formula:

[0102]

[0103] Where k1, k2, k3, k4, k5, and k6 represent the weighting coefficients of the corresponding abnormal indicator values, and all k1, k2, k3, k4, k5, and k6 are greater than or equal to 0 and less than or equal to 1. Furthermore, the weighting coefficient for each abnormal indicator value can be arbitrarily set according to the user's actual needs. For example, the weighting coefficient for abnormal indicator values ​​that users pay more attention to can be set relatively large, while the weighting coefficient for abnormal indicator values ​​that users pay less attention to can be set relatively small. In practical applications, for example, there are various distinct regions in a panel IC, such as low-frequency regions, high-frequency regions, and particle regions. For anomaly detection in a single type of region, determining one abnormal indicator value can describe its degree of abnormality. Therefore, the weighting coefficient of that abnormal indicator value can be set relatively large, for example, to 1, while the weighting coefficients of other abnormal indicator values ​​can be set relatively small, for example, all to 0. However, when simultaneously detecting anomalies in multiple different regions of an integrated circuit, a single anomaly index value cannot adequately determine the degree of anomaly in each region. Therefore, a comprehensive anomaly index value for each region can be determined using the formula described above, and the weighting coefficients for each anomaly index value can be set relatively evenly. For example, k1 = k5 = k6 = 0.3, k2 = k3 = 0.1, and k4 = 0.2. Thus, the method for determining the comprehensive anomaly index value of the above-described anomaly foreground can quickly and accurately determine the degree of anomaly in various anomaly foregrounds.

[0104] In another embodiment, multiple quantified metrics from the first and second gray-level histograms can be input into a trained decision tree model. The model can then be trained to output a comprehensive anomaly index value for the anomalous foreground. In this approach, the decision tree model can regress a more ideal comprehensive anomaly index value for the anomalous foreground. Furthermore, because the decision tree model presents a non-linear model structure, it provides a better user experience.

[0105] After obtaining the comprehensive anomaly index value of the abnormal foreground using the above method, this comprehensive anomaly index value can be compared with a preset index threshold. Based on the comparison result, the final detection result can be determined. For example, if the comprehensive anomaly index value is greater than the preset index threshold, the abnormal foreground of the current image to be detected can be determined as the final abnormal region. For instance, if the comprehensive anomaly index value of the abnormal foreground in the current image to be detected is determined to be 0.6 using the above formula, and the preset index threshold is 0.5, then the abnormal foreground of the current image to be detected output by the detection model can be determined as the identified abnormal region, and this abnormal region can be output as the final detection result. Therefore, the comprehensive anomaly index of the abnormal foreground can be determined through post-processing of the abnormal foreground, thereby determining the final detection result of the anomaly detection.

[0106] According to the above scheme, in the post-processing stage of anomaly detection, the precise localization region of the normal template in the normal template image can be determined. Statistical analysis of grayscale histograms is then performed on both the anomalous foreground and the precise localization region of the normal template, yielding their respective statistical analysis results. By comparing the statistical analysis results of the two, multiple anomaly index values ​​of the anomalous foreground are determined, and a comprehensive anomaly index value of the anomalous foreground is obtained using linear or non-linear methods. Finally, the final anomaly detection result can be determined based on the comprehensive anomaly index value. In this scheme, the degree of anomalousness of the anomalous foreground is presented to the user intuitively and accurately through the quantified comprehensive anomaly index value, significantly improving detection accuracy while also providing a better user experience.

[0107] Figure 5 A flowchart illustrating an anomaly detection method according to another embodiment of this application is shown. Figure 5As shown, the image to be detected, including the target object, can be input into a trained detection model, which outputs a rectangular bounding box of the abnormal region in the image. The location of the abnormal region is, for example, (x0, y0, w, h). Based on the center (x0, y0) of the abnormal region, an expanded abnormal region can be obtained, with its outer edge located, for example, (x0, y0, 2w, 2h). The location of the coarse localization region of the normal template image can also be obtained, with its outer edge located, for example, (x0, y0, 4w, 4h). Then, any suitable feature point extraction method can be used to extract the first feature point of the expanded abnormal region and the second feature point of the coarse localization region of the normal template. The first and second feature points can be matched using RANSAC matching to obtain the feature point matching result. Then, an affine transformation can be performed on the normal template image based on the feature point matching result to accurately align the normal template image and the image to be detected. Finally, the fine localization region of the normal template can be accurately located based on the location of the abnormal region. Simultaneously, the mapping relationship between pixels in the normal template fine-localization area and pixels in the abnormal area can be determined based on the feature point matching results. The brightness of pixels in the normal template fine-localization area is then calibrated according to this mapping relationship to ensure that the average brightness of the calibrated template fine-localization area is the same as that of the abnormal area. Next, the gray-level histograms and gray-level histogram integral curves of the abnormal area and the calibrated template fine-localization area can be obtained separately. Multiple quantifiable anomaly indicators can be determined by statistically analyzing the differences between the gray-level histograms and gray-level histogram integral curves of the two areas. A decision tree model or a formula for calculating the comprehensive anomaly indicator value can be used to determine the comprehensive anomaly indicator value of the abnormal foreground. Finally, the comprehensive anomaly indicator value can be compared with a preset threshold, and the final detection result of the image to be detected can be determined based on the comparison result.

[0108] According to a third aspect of this application, a foreground extraction system is provided. Figure 6 A schematic block diagram of a foreground extraction system 600 according to an embodiment of this application is shown. As shown, the foreground extraction system 600 includes:

[0109] The acquisition module 610 is used to acquire the foreground extension area located near the foreground area in the image to be extracted based on the location information of the foreground area in the image to be extracted;

[0110] The first determining module 620 is used to determine the template coarse positioning area in the template image based on the position information of the foreground area in the image to be extracted, wherein the template coarse positioning area includes the area corresponding to the foreground expansion area;

[0111] The matching module 630 is used to perform feature matching between the foreground expansion area and the template coarse positioning area to obtain the first matching result;

[0112] The second determining module 640 is used to determine, based on the first matching result, the template fine-positioning region in the template image corresponding to the foreground region; and

[0113] The extraction module 650 is used to extract the foreground based on at least the pixel value of each pixel in the foreground area and the pixel value of the corresponding pixel in the template fine positioning area.

[0114] According to the fourth aspect of this application, an anomaly detection system is provided. Figure 7 A schematic block diagram of an anomaly detection system 700 according to an embodiment of this application is shown. As shown, the system 700 includes a detection module 710 for inputting an image to be detected into a trained detection model to obtain abnormal foregrounds in the image. The detection model is trained using labeled training sample images, which are composite images of normal sample images and abnormal foregrounds extracted using the foreground extraction method 100 described above.

[0115] According to a fifth aspect of this application, an electronic device is also provided. Figure 8 A schematic block diagram of an electronic device 800 according to an embodiment of this application is shown. As shown, the electronic device 800 includes a processor 810 and a memory 820, wherein the memory 820 stores computer program instructions, which are executed by the processor 810 to perform the foreground extraction method 100 or the anomaly detection method 400 described above.

[0116] According to a sixth aspect of this application, a storage medium is also provided, on which program instructions are stored, which, when executed, are used to perform the foreground extraction method 100 or the anomaly detection method 300 described above. The storage medium may, for example, include a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0117] Those skilled in the art can understand the specific implementation schemes of the foreground extraction system 600, the anomaly detection system 700, the electronic device 800, and the storage medium by reading the above descriptions of the foreground extraction method 100 and the anomaly detection method 400. For the sake of brevity, they will not be described in detail here.

[0118] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0119] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0121] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0122] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0123] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0124] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0125] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the foreground extraction system and anomaly detection system according to embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0126] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0127] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A foreground extraction method, characterized in that, Including: According to the position information of the foreground area in the image to be extracted, obtain a foreground expansion area near the foreground area in the image to be extracted, where the foreground expansion area is used to indicate the area expanded from the edge of the foreground area; According to the position information of the foreground area in the image to be extracted, determine a template rough positioning area in the template image, where the template rough positioning area includes an area corresponding to the foreground expansion area, and the outer edge size of the template rough positioning area is larger than the outer edge size of the foreground expansion area, and the image to be extracted and the template image are images of the same target object; Perform feature matching between the foreground expansion area and the template rough positioning area to obtain a first matching result; According to the first matching result, determine a template fine positioning area in the template image corresponding to the foreground area; and Extract the foreground at least according to the pixel value of each pixel in the foreground area and the pixel value of the corresponding pixel in the template fine positioning area, where the foreground is used to synthesize a training sample image.

2. The foreground extraction method as described in claim 1, characterized in that, The extracting the foreground at least according to the pixel value of each pixel in the foreground area and the pixel value of the corresponding pixel in the template fine positioning area includes: Adjust the brightness of the template fine positioning area or the foreground area so that after adjustment, the template fine positioning area and the foreground area are at the same brightness level; and Extract the foreground by comparing the pixel value of each pixel in the foreground area at the same brightness level with the pixel value of the corresponding pixel in the template fine positioning area.

3. The foreground extraction method as described in claim 2, characterized in that, The adjusting the brightness of the template fine positioning area or the foreground area includes: Based on the first matching result of the foreground expansion area and the template rough positioning area, determine a brightness mapping relationship; Use the brightness mapping relationship to adjust the brightness of the template fine positioning area or the foreground area.

4. The foreground extraction method according to any one of claims 1 to 3, characterized in that, The foreground expansion area includes an annular area surrounding the foreground area on all sides.

5. The foreground extraction method as described in claim 4, characterized in that, The width of the outer edge of the foreground expansion area is equal to n1 times the width of the foreground area, and the height of the outer edge of the foreground expansion area is equal to m1 times the height of the foreground area; The width of the outer edge of the template rough positioning area is equal to n2 times the width of the foreground area, and the height of the outer edge of the template rough positioning area is equal to m2 times the height of the foreground area, where n1 < n2 and / or m1 < m2.

6. The foreground extraction method according to any one of claims 1 to 3, characterized in that, The method further includes: performing noise filtering on the foreground to obtain the filtered foreground.

7. An anomaly detection method, characterized in that, Including: Input the image to be detected into a trained detection model to obtain an abnormal foreground in the image to be detected, where the detection model is trained by a labeled training sample image, and the labeled training sample image is a synthetic image of a normal sample image and an abnormal foreground extracted by the foreground extraction method according to any one of claims 1 to 6.

8. The anomaly detection method as described in claim 7, characterized in that, The method further includes: Construct an abnormal foreground dataset based on the abnormal foreground; Randomly select a current abnormal foreground from the abnormal foreground dataset; Perform a first brightness adjustment on the current abnormal foreground with a first probability and synthesize it with the corresponding normal sample image to obtain a positive sample image that is greater than or equal to a first brightness threshold and includes an abnormal label; The current abnormal foreground is adjusted for brightness with a second probability and synthesized with the corresponding normal sample image to obtain a negative sample image that is less than the second brightness threshold and includes normal labels. The labeled training sample images include the positive sample images and the negative sample images, the first probability is greater than the second probability, the second brightness threshold is equal to the first brightness threshold multiplied by a preset ratio, and the preset ratio is less than 0.

5.

9. The anomaly detection method as described in claim 7, characterized in that, After acquiring the abnormal foreground in the image to be detected, the method further includes: Based at least on the position of the abnormal foreground in the image to be detected, a precise localization region of the normal template in the normal template image corresponding to the abnormal foreground is determined; Gray-scale histogram statistical analysis was performed on the abnormal foreground and the precise positioning area of ​​the normal template, and the statistical analysis results were compared to determine the abnormal index value of the abnormal foreground.

10. The anomaly detection method as described in claim 9, characterized in that, The step of determining the precise localization region of the normal template in the normal template image corresponding to the abnormal foreground, based at least on the position of the abnormal foreground in the image to be detected, includes: Based on the location information of the abnormal foreground in the image to be detected, an abnormal outward expansion region located around the abnormal foreground is obtained in the image to be detected. Based on the location information of the abnormal outward expansion area in the image to be detected, a normal template coarse positioning area in the normal template image is determined, wherein the normal template coarse positioning area includes the region corresponding to the abnormal outward expansion area; Feature matching is performed on the abnormal outward expansion region and the normal template coarse localization region to obtain a second matching result; Based on the second matching result, the precise localization region of the normal template corresponding to the abnormal foreground in the normal template image is determined.

11. The anomaly detection method as described in claim 9 or 10, characterized in that, Before performing grayscale histogram statistical analysis on the abnormal foreground and the precise localization area of ​​the normal template respectively, the method further includes: The brightness of the normal template fine positioning area or the abnormal foreground is adjusted so that the template fine positioning area and the abnormal foreground are at the same brightness level after adjustment.

12. The anomaly detection method as described in claim 9 or 10, characterized in that, The step of performing grayscale histogram statistical analysis on the abnormal foreground and the precise localization area of ​​the normal template, and comparing the statistical analysis results to determine the abnormal index value of the abnormal foreground, includes: The first grayscale histogram of the abnormal foreground and the second grayscale histogram of the precise positioning area of ​​the normal template are obtained respectively. Statistical analysis based on the first gray-level histogram and the second gray-level histogram determines multiple abnormal index values ​​of the abnormal foreground; The comprehensive abnormality index value of the abnormal prospect is determined based on the multiple abnormality index values.

13. A foreground extraction system, characterized in that, include: The acquisition module is used to acquire a foreground extension area located near the foreground area in the image to be extracted based on the location information of the foreground area in the image to be extracted, wherein the foreground extension area is used to indicate the area extending from the edge of the foreground area; The first determining module is used to determine a template coarse positioning region in the template image based on the position information of the foreground region in the image to be extracted, wherein the template coarse positioning region includes a region corresponding to the foreground expansion region, the outer edge size of the template coarse positioning region is larger than the outer edge size of the foreground expansion region, and the image to be extracted and the template image are images of the same target object; The matching module is used to perform feature matching between the foreground expansion area and the template coarse positioning area to obtain a first matching result; The second determining module is configured to determine, based on the first matching result, a template fine-positioning region in the template image corresponding to the foreground region; and An extraction module is used to extract the foreground based on at least the pixel value of each pixel in the foreground region and the pixel value of the corresponding pixel in the template fine-positioning region, wherein the foreground is used to synthesize a training sample image.

14. An anomaly detection system, characterized in that, include: A detection module is used to input the image to be detected into a trained detection model to obtain abnormal foregrounds in the image to be detected. The detection model is trained from labeled training sample images, which are composite images of normal sample images and abnormal foregrounds extracted using the foreground extraction method according to any one of claims 1 to 6.

15. An electronic device comprising a processor and a memory, wherein, The memory stores computer program instructions, which, when executed by the processor, are used to perform the foreground extraction method as described in any one of claims 1 to 6 or the anomaly detection method as described in any one of claims 7 to 12.

16. A storage medium storing program instructions that, when executed, perform a foreground extraction method as claimed in any one of claims 1 to 6 or an anomaly detection method as claimed in any one of claims 7 to 12.

Images