Semantic segmentation-based out-of-distribution abnormal target detection method

Abstract

The invention relates to an out-of-distribution abnormal target detection method based on semantic segmentation, and the method comprises the following steps: obtaining the semantic information of each pixel point of a to-be-detected image based on a semantic segmentation neural network; selecting the pixel points belonging to the same category, and completing the preliminary segmentation of the to-be-detected image according to the category to which each pixel point is judged to belong; performing secondary segmentation on the preliminary segmentation result by using a connected domain algorithm; if the number of pixels contained in the image blocks obtained through secondary segmentation exceeds a preset threshold value, segmentation is carried out again according to the edge detection result of the original to-be-detected image, and a plurality of segmented image blocks serving as anomaly detection minimum units are obtained; according to semantic information of each pixel point, performing anomaly scoring on each delimited image block by using information entropy; and outputting the image blocks of which the anomalies are higher than the judgment threshold as anomaly detection results. Compared with the prior art, the method has the advantages that the semantic continuity of the detection result is ensured, and the out-of-distribution abnormal target can be recognized with higher precision.

Description

Technical Field

[0001] The present invention relates to anomaly detection technology, and in particular to an out-of-distribution abnormal target detection method based on semantic segmentation. Background Art

[0002] In recent years, various visual perception algorithms based on neural networks and deep learning have achieved great success. However, these visual perception algorithms can only detect pre-defined and pre-classified targets that already exist in the given data set during training. For targets that do not belong to the types in the training set, the neural network will forcibly classify them into one of the pre-set known classes, which may bring safety issues. For example, human-powered tricycles are obviously rare in a training data set from urban areas, but they may be very common in actual driving scenarios in rural areas. At this time, misidentifying human-powered tricycles as other categories may lead to traffic safety risks. Therefore, it is necessary to perform anomaly detection on out-of-distribution abnormal targets outside the preset categories based on the recognition results of the visual perception algorithm, which will help improve the information perception ability of the information perception system to the overall environmental situation and provide corresponding information support and technical support for the users of the visual detection system to make the final decision.

[0003] CN113989305A discloses a method for semantic segmentation of a target, including: obtaining an image to be detected including at least one target; inputting the image to be detected into a target detection network to obtain a detection frame of at least one target; setting a growth region and an initial seed point according to the detection frame, and obtaining a segmentation contour of the corresponding target according to the regional growth result of each initial seed point; inputting the image to be detected into the predicted contour of each target of the target segmentation network; obtaining a semantic segmentation result of the corresponding target according to the union region of the segmentation contour and the predicted contour of each identical target. This method solves the problem of difficult labeling in semantic segmentation by using target detection to generate a detection frame of the target, and determines the growth region and the initial seed point according to the detection frame, and uses the regional growing algorithm to segment the target to obtain a more accurate segmentation contour of the target, and uses the target segmentation network to obtain the predicted contour of the target, and obtains the semantic segmentation result according to the segmentation contour and the segmentation contour of the identical target. However, this method requires the use of two networks, which will cause additional computational overhead, and does not consider semantic coherence, resulting in the accuracy of the detection result being affected. Summary of the invention

[0004] The purpose of the present invention is to provide a method for detecting out-of-distribution abnormal targets based on semantic segmentation, which performs out-of-distribution abnormal target detection on the image to be detected based on the semantic segmentation result, and completes the task of identifying out-of-distribution abnormal targets in the image to be detected that do not belong to the preset categories in the training set, without the need for additional neural network training and excessive computing performance, and has better comprehensive detection performance.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for detecting out-of-distribution abnormal targets based on semantic segmentation comprises the following steps:

[0007] Based on the semantic segmentation neural network, the semantic information of each pixel of the image to be detected is obtained, wherein the semantic information of each pixel includes the category to which each pixel belongs and the probability of belonging to the category;

[0008] Select pixels belonging to the same category, and complete the preliminary segmentation of the image to be detected according to the category to which each pixel is determined to belong;

[0009] Use the connected domain algorithm to perform secondary segmentation on the initial segmentation results;

[0010] If the number of pixels contained in the image block obtained by the secondary segmentation exceeds a preset threshold, the image is segmented again according to the edge detection result of the original image to be detected to obtain multiple image blocks as the smallest units for abnormality detection after segmentation;

[0011] According to the semantic information of each pixel, the information entropy is used to score the abnormality of each demarcated image block;

[0012] The image blocks with abnormality higher than the judgment threshold are output as abnormality detection results.

[0013] The structure of the semantic segmentation neural network model is the DeepLab v3+ network structure.

[0014] The steps of the connected domain algorithm are as follows:

[0015] Step 301: Create an empty stack for storing pixels;

[0016] Step 302: Select a certain pixel point as the pixel point to be detected;

[0017] Step 303: Observe whether the pixel to be detected is marked. If the pixel to be detected is marked, return to step 302 and reselect a pixel as the pixel to be detected;

[0018] Step 304: if the pixel to be detected is not marked, mark it, and push other pixels connected to the pixel to be detected into a stack;

[0019] Step 305: Observe whether the stack is empty. If the stack is empty, modify the marking method of the pixels to complete the detection of a set of connected domains, and return to step 301;

[0020] Step 306: if the stack is not empty, the pixel pushed in step 304 is popped out of the stack, and the algorithm is continued to be executed on the pixel popped out of the stack from step 303, wherein, when the pixel popped out of the stack has been marked in advance, step 303 is executed on the next pixel to be popped out of the stack;

[0021] Step 307: When the stack is empty and all pixels are marked, the connected domain detection is completed, and pixels with the same mark are output as the connected domains obtained by the detection.

[0022] In step 304, the criteria for determining whether a pixel point is connected to a given pixel point to be detected include 4-connectivity determination and 8-connectivity determination.

[0023] The 4-connectivity determination criteria are as follows:

[0024]

[0025] The 8-connectivity determination criteria are as follows:

[0026]

[0027] Among them, the pixel point marked as 2 is a given pixel point to be detected, and the pixel point at the position marked as 1 in the determination matrix is ​​considered to be connected with the pixel point to be detected.

[0028] The steps of re-segmentation are as follows:

[0029] Step 401: using an edge detection operator, calculating the score of each pixel as an edge pixel of the target object to be detected;

[0030] Step 402: for the image block to be segmented, consider the pixels located at its boundary, select the pixel with the highest edge pixel score, record it as the first pixel, and mark it;

[0031] Step 403: for the first pixel point obtained in step 402, consider all the pixels connected to it, select the second pixel point with the highest edge pixel score, and mark it;

[0032] Step 404: for the second pixel point obtained in step 403, observe whether it is located at the boundary of the image block to be segmented. If it is not at the boundary, execute step 403 for the second pixel point;

[0033] Step 405: For the second pixel point obtained in step 403, if it is at the boundary, all the marked pixel points are used as the boundary to complete the re-segmentation of the image block to be segmented.

[0034] In step 401, the edge detection operator used to calculate each pixel point as the edge pixel point of the target object to be detected includes a Sobel operator, a Roberts operator or a Prewitt operator.

[0035] When the operator used is the Sobel operator, its horizontal edge detection matrix operator is:

[0036]

[0037] Its vertical edge detection matrix operator is:

[0038]

[0039] Use the following formula to perform planar convolution calculation on the image to be detected T, and use the result as the edge pixel score G:

[0040]

[0041] in, Represents convolution.

[0042] The calculation method of the information entropy is as follows:

[0043]

[0044] Among them, H is the information entropy of the pixel to be evaluated, p i The probability that the pixel belongs to the preset category i is obtained by using the semantic segmentation neural network.

[0045] The abnormality is represented by the average information entropy of each image block, and the calculation method of the average information entropy is as follows:

[0046]

[0047] in, is the average information entropy of the segmented image blocks, H i is the information entropy calculated for pixel i.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. Based on the semantic segmentation neural network, the present invention uses the neural network and the original image information to predict the probability that the target to be detected does not belong to a given category, which to a certain extent avoids the security issues that may be caused by being limited by the categories set during training.

[0050] 2. The present invention directly uses the detection results of the neural network as the basis for determining whether there is an abnormality or not, without the need to design an additional neural network for training. To a certain extent, it avoids the problem that traditional anomaly detection methods require additional neural networks for training and detection, thereby causing additional computing overhead and ultimately leading to the problem of long detection time.

[0051] 3. The present invention uses pixel blocks with the same semantics as the minimum detection unit, which improves the semantic coherence of the anomaly detection result, thereby improving the accuracy of the final anomaly detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 It is a schematic diagram of a process of completing segmentation of an image block as a minimum unit for out-of-distribution anomaly detection during the implementation of the present invention;

[0054] Figure 3 It is an algorithm flow chart of the connected domain detection algorithm used in the implementation process of the present invention;

[0055] Figure 4 The flowchart is an algorithm for re-segmenting image blocks using image edge information during the implementation of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0057] This embodiment provides a method for detecting out-of-distribution abnormal targets based on semantic segmentation. Figure 1 As shown, the following steps are included:

[0058] Step 1: Based on the semantic segmentation neural network, the semantic information of each pixel of the image to be detected is obtained, wherein the semantic information of each pixel includes the category to which each pixel belongs and the probability of belonging to the category.

[0059] In this embodiment, the structure of the semantic segmentation neural network model is the DeepLab v3+ network structure.

[0060] like Figure 2 As shown, step 2 to step 4 are the process of completing the segmentation of the image block as the smallest unit for out-of-distribution anomaly detection.

[0061] Step 2: Select pixels belonging to the same category and complete the preliminary segmentation of the image to be detected according to the category to which each pixel is determined to belong.

[0062] Step 3: Use the connected domain algorithm to perform secondary segmentation on the initial segmentation results.

[0063] like Figure 3 As shown, the steps of the connected domain algorithm are as follows:

[0064] Step 301: Create an empty stack for storing pixels;

[0065] Step 302: Select a certain pixel point as the pixel point A to be detected;

[0066] Step 303: Observe whether the pixel point A to be detected is marked. If the pixel point to be detected is marked, return to step 302 and reselect a pixel point as the pixel point A to be detected;

[0067] Step 304: if the pixel A to be detected is not marked, mark it, and push other pixels connected to the pixel A to be detected into a stack;

[0068] Step 305: Observe whether the stack is empty. If the stack is empty, modify the marking method of the pixels to complete the detection of a set of connected domains, and return to step 301;

[0069] Step 306: if the stack is not empty, the pixel point B pushed in step 304 is popped out of the stack, and the algorithm is continued to be executed on the pixel point B that is popped out of the stack from step 303. When the pixel point B that is popped out of the stack has been marked in advance, step 303 is executed on the next pixel point C that is popped out of the stack.

[0070] Step 307: When the stack is empty and all pixels are marked, the connected domain detection is completed, and pixels with the same mark are output as the connected domains obtained by the detection.

[0071] In step 304, the criteria for determining whether a pixel point is connected to the given pixel point A to be detected include 4-connectivity determination and 8-connectivity determination.

[0072] Among them, the 4-connectivity judgment criteria are as follows:

[0073]

[0074] 8 The connectivity determination criteria are as follows:

[0075]

[0076] Among them, the pixel point marked as 2 is a given pixel point to be detected, and the pixel point at the position marked as 1 in the determination matrix is ​​considered to be connected with the pixel point to be detected.

[0077] Step 4: If the number of pixels in the image block obtained by the secondary segmentation exceeds a preset threshold, the image is segmented again according to the edge detection result of the original image to be detected to obtain multiple image blocks as the smallest units for abnormality detection after segmentation.

[0078] Considering that the number of preset categories in the neural network training set may be small, the image blocks segmented by the connected domain algorithm may contain too many pixels, which will greatly damage the accuracy of the anomaly detection results. Therefore, for image blocks containing too many pixels, the edge detection algorithm is used to complete the re-segmentation of this image block. Figure 4 As shown, the steps for splitting again are as follows:

[0079] Step 401: using an edge detection operator, calculating the score of each pixel as an edge pixel of the target object to be detected;

[0080] Step 402: for the image block to be segmented, consider the pixels located at its boundary, select the pixel with the highest edge pixel score, record it as the first pixel D, and mark it;

[0081] Step 403: for the first pixel point D obtained in step 402, consider all the pixels connected to it, select the second pixel point E with the highest edge pixel score, and mark it;

[0082] Step 404: for the second pixel point E obtained in step 403, observe whether it is located at the boundary of the image block to be segmented. If it is not at the boundary, execute step 403 for the second pixel point E;

[0083] Step 405: For the second pixel point E obtained in step 403, if it is at the boundary, all the marked pixel points are used as boundaries to complete the re-segmentation of the image block to be segmented.

[0084] In step 401, the edge detection operator used to calculate each pixel point as the edge pixel point of the target object to be detected includes but is not limited to the Sobel operator, the Roberts operator or the Prewitt operator.

[0085] When the operator used is the Sobel operator, its horizontal edge detection matrix operator is:

[0086]

[0087] Its vertical edge detection matrix operator is:

[0088]

[0089] Use the following formula to perform planar convolution calculation on the image to be detected T, and use the result as the edge pixel score G:

[0090]

[0091] in, Represents convolution.

[0092] Step 5: Based on the semantic information of each pixel, use information entropy to score the abnormality of each demarcated image block.

[0093] The calculation method of information entropy is as follows:

[0094]

[0095] Among them, H is the information entropy of the pixel to be evaluated, p i The probability that the pixel belongs to the preset category i is obtained by using the semantic segmentation neural network.

[0096] The degree of abnormality is represented by the average information entropy of each image block, and the calculation method of the average information entropy is as follows:

[0097]

[0098] in, is the average information entropy of the segmented image blocks, H i is the information entropy calculated for pixel i.

[0099] Step 6: Output the image blocks whose abnormality is higher than the judgment threshold as the abnormality detection result.

[0100] Specifically, when determining whether the segmented image block is abnormal, a threshold determination method is used. A determination threshold is set according to the category to which the neural network determines the pixel points in the image block to be detected belong. When the average information entropy of this image block exceeds this threshold, this image block is determined to be abnormal. When the average information entropy of this image block is lower than this threshold, it is determined that this image block is indeed one of the preset categories in the training set, and the detection result of the neural network is directly output.

[0101] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for detecting out-of-distribution abnormal targets based on semantic segmentation, characterized in that: The following steps are involved: Based on the semantic segmentation neural network, the semantic information of each pixel of the image to be detected is obtained, wherein the semantic information of each pixel includes the category to which each pixel belongs and the probability of belonging to the category; Select pixels belonging to the same category, and complete the preliminary segmentation of the image to be detected according to the category to which each pixel is determined to belong; Use the connected domain algorithm to perform secondary segmentation on the initial segmentation results; If the number of pixels contained in the image block obtained by the secondary segmentation exceeds a preset threshold, the image is segmented again according to the edge detection result of the original image to be detected to obtain multiple image blocks as the smallest units for abnormality detection after segmentation; According to the semantic information of each pixel, the information entropy is used to score the abnormality of each demarcated image block; The image blocks with abnormality higher than the judgment threshold are output as abnormality detection results.

2. According to the method of claim 1, the method is characterized in that: The structure of the semantic segmentation neural network model is the DeepLab v3+ network structure.

3. The method for detecting out-of-distribution abnormal targets based on semantic segmentation according to claim 1, characterized in that: The steps of the connected domain algorithm are as follows: Step 301: Create an empty stack for storing pixels; Step 302: Select a certain pixel point as the pixel point to be detected; Step 303: Observe whether the pixel to be detected is marked. If the pixel to be detected is marked, return to step 302 and reselect a pixel as the pixel to be detected; Step 304: if the pixel to be detected is not marked, mark it, and push other pixels connected to the pixel to be detected into a stack; Step 305: Observe whether the stack is empty. If the stack is empty, modify the marking method of the pixels to complete the detection of a set of connected domains, and return to step 301; Step 306: if the stack is not empty, the pixel pushed in step 304 is popped out of the stack, and the algorithm is continued to be executed on the pixel popped out of the stack from step 303, wherein, when the pixel popped out of the stack has been marked in advance, step 303 is executed on the next pixel to be popped out of the stack; Step 307: When the stack is empty and all pixels are marked, the connected domain detection is completed, and pixels with the same mark are output as the connected domains obtained by the detection.

4. The method for detecting out-of-distribution abnormal targets based on semantic segmentation according to claim 3, characterized in that: In step 304, the criteria for determining whether a pixel point is connected to a given pixel point to be detected include 4-connectivity determination and 8-connectivity determination.

5. The method for detecting out-of-distribution abnormal targets based on semantic segmentation according to claim 4, characterized in that: The 4-connectivity determination criteria are as follows: The 8-connectivity determination criteria are as follows: Among them, the pixel point marked as 2 is a given pixel point to be detected, and the pixel point at the position marked as 1 in the determination matrix is ​​considered to be connected with the pixel point to be detected.

6. The method for detecting out-of-distribution abnormal targets based on semantic segmentation according to claim 1, characterized in that: The steps of re-segmentation are as follows: Step 401: using an edge detection operator, calculating the score of each pixel as an edge pixel of the target object to be detected; Step 402: for the image block to be segmented, consider the pixels located at its boundary, select the pixel with the highest edge pixel score, record it as the first pixel, and mark it; Step 403: for the first pixel point obtained in step 402, consider all the pixels connected to it, select the second pixel point with the highest edge pixel score, and mark it; Step 404: for the second pixel point obtained in step 403, observe whether it is located at the boundary of the image block to be segmented. If it is not at the boundary, execute step 403 for the second pixel point; Step 405: For the second pixel point obtained in step 403, if it is at the boundary, all the marked pixel points are used as the boundary to complete the re-segmentation of the image block to be segmented.

7. The method for detecting out-of-distribution abnormal targets based on semantic segmentation according to claim 6, characterized in that: In step 401, the edge detection operator used to calculate each pixel point as the edge pixel point of the target object to be detected includes a Sobel operator, a Roberts operator or a Prewitt operator.

8. The method for detecting out-of-distribution abnormal targets based on semantic segmentation according to claim 7, characterized in that: When the operator used is the Sobel operator, its horizontal edge detection matrix operator is: Its vertical edge detection matrix operator is: Use the following formula to perform planar convolution calculation on the image to be detected T, and use the result as the edge pixel score G: in, Represents convolution.

9. The method for detecting out-of-distribution abnormal targets based on semantic segmentation according to claim 1, characterized in that: The calculation method of the information entropy is as follows: Among them, H is the information entropy of the pixel to be evaluated, p i The probability that the pixel belongs to the preset category i is obtained by using the semantic segmentation neural network.

10. The method for detecting out-of-distribution abnormal targets based on semantic segmentation according to claim 9, characterized in that: The abnormality is represented by the average information entropy of each image block, and the calculation method of the average information entropy is as follows: in, is the average information entropy of the segmented image blocks, H i is the information entropy calculated for pixel i.

Images