Image segmentation anomaly detection method, device, equipment, storage medium and product

Abstract

The embodiment of the present application provides an image segmentation anomaly detection method, device, equipment, storage medium and product. The technical scheme provided by the embodiment of the present application determines the spatial domain anomaly probability of the current segmentation output mask in the spatial domain according to the pixel proportion of the probability value in the set probability range in the current segmentation output mask, determines the time domain anomaly probability of the current segmentation output mask in the time domain according to the color value difference between the current target frame image and the last target frame image and the probability value difference between the current segmentation output mask and the last segmentation output mask, and determines the anomaly detection result of the current segmentation output mask according to the spatial domain anomaly probability and the time domain anomaly probability. The anomaly degree of the image segmentation result can be evaluated in the spatial domain and the time domain, whether the image segmentation result is abnormal can be accurately judged, the manually screened abnormal segmentation result is not needed, and the image segmentation anomaly detection efficiency is effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to image segmentation anomaly detection methods, apparatus, devices, storage media, and products. Background Technology

[0002] Image segmentation is a common method in image processing that annotates specific targets in an image at the pixel level. The input to image segmentation is typically a natural image (e.g., a three-channel RGB image or a single-channel YUV image), and the output is a mask with the same dimensions as the natural image. The mask usually has only one channel, and the probability values ​​range from [0, 255]. Within the pixels of the mask, a higher probability value indicates a greater probability that the location in the natural image represents a specific target.

[0003] The performance of deep learning-based image segmentation models is highly dependent on the quality of the dataset. During model use, there may be instances where the model performs poorly in certain scenarios. It is necessary to supplement the dataset with anomaly detection results for similar scenarios to improve the performance of the deep learning model. Currently, anomaly detection results are typically collected through manual screening, meaning that the image segmentation results are determined by human judgment to determine whether they are abnormal. This method has low efficiency in image segmentation anomaly detection. Summary of the Invention

[0004] This application provides methods, apparatus, devices, storage media, and products for image segmentation anomaly detection, in order to solve the technical problem of low efficiency in image segmentation anomaly detection in related technologies and effectively improve the efficiency of image segmentation anomaly detection.

[0005] In a first aspect, embodiments of this application provide an image segmentation anomaly detection method, comprising:

[0006] The current target bounding box image, the previous target bounding box image, the current segmentation output mask, and the previous segmentation output mask are obtained. The current target bounding box image and the current segmentation output mask are obtained by performing image segmentation on the current input image using a set image segmentation model. The previous target bounding box image and the previous segmentation output mask are obtained by performing image segmentation on the previous input image using the image segmentation model.

[0007] Based on the percentage of pixels whose probability values ​​are within a set probability range in the current segmentation output mask, the spatial anomaly probability of the current segmentation output mask in the spatial domain is determined.

[0008] Based on the color value difference between the current target bounding box image and the previous target bounding box image, and the probability value difference between the current segmentation output mask and the previous segmentation output mask, the temporal anomaly probability of the current segmentation output mask in the time domain is determined.

[0009] The anomaly detection result of the current segmentation output mask is determined based on the spatial anomaly probability and the temporal anomaly probability.

[0010] In a second aspect, embodiments of this application provide an image segmentation anomaly detection device, including an image acquisition module, a spatial domain detection module, a temporal domain detection module, and an anomaly determination module, wherein:

[0011] The image acquisition module is configured to acquire the current target bounding box image, the previous target bounding box image, the current segmentation output mask, and the previous segmentation output mask. The current target bounding box image and the current segmentation output mask are obtained by performing image segmentation on the current input image using a set image segmentation model. The previous target bounding box image and the previous segmentation output mask are obtained by performing image segmentation on the previous input image using the image segmentation model.

[0012] The spatial domain detection module is configured to determine the spatial domain anomaly probability of the current segmentation output mask in the spatial domain based on the percentage of pixels whose probability values ​​are within a set probability range in the current segmentation output mask.

[0013] The temporal detection module is configured to determine the temporal anomaly probability of the current segmentation output mask in the temporal domain based on the color value difference between the current target bounding box image and the previous target bounding box image, and the probability value difference between the current segmentation output mask and the previous segmentation output mask.

[0014] The anomaly determination module is configured to determine the anomaly detection result of the current segmentation output mask based on the spatial anomaly probability and the temporal anomaly probability.

[0015] In a third aspect, embodiments of this application provide an image segmentation anomaly detection device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the image segmentation anomaly detection method as described in the first aspect.

[0018] In a fourth aspect, embodiments of this application provide a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the image segmentation anomaly detection method as described in the first aspect.

[0019] In a fifth aspect, embodiments of this application provide a computer program product comprising a computer program stored in a computer-readable storage medium, wherein at least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the image segmentation anomaly detection method as described in the first aspect.

[0020] This application embodiment determines the spatial anomaly probability of the current segmentation output mask in the spatial domain based on the proportion of pixels with probability values ​​within a set probability range in the current segmentation output mask, and determines the temporal anomaly probability of the current segmentation output mask in the temporal domain based on the color value difference between the current target box image and the previous target box image, as well as the probability value difference between the current segmentation output mask and the previous segmentation output mask. Based on the spatial and temporal anomaly probabilities, the anomaly detection result of the current segmentation output mask is determined. This allows for the evaluation of the degree of anomaly in the image segmentation results in both the spatial and temporal domains, accurately determining whether the image segmentation results are abnormal, eliminating the need for manual screening of abnormal segmentation results, and effectively improving the efficiency of image segmentation anomaly detection. Attached Figure Description

[0021] Figure 1 This is a flowchart of an image segmentation anomaly detection method provided in an embodiment of this application;

[0022] Figure 2 This is a flowchart of another image segmentation anomaly detection method provided in the embodiments of this application;

[0023] Figure 3 This is a schematic diagram of the structure of an image segmentation anomaly detection device provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of an image segmentation anomaly detection device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but additional steps not included in the drawings may also be present. The above processes can correspond to methods, functions, procedures, subroutines, subroutines, etc.

[0026] The image segmentation anomaly detection method provided in this application can be applied to the image segmentation process of video, detecting anomalies in the segmentation results. It aims to determine the anomaly detection result of the current segmentation output mask based on the spatial and temporal anomaly probabilities, evaluating the degree of anomaly in the image segmentation results in both spatial and temporal domains. This accurately determines whether the image segmentation results are abnormal, eliminating the need for manual screening of abnormal segmentation results and effectively improving the efficiency of image segmentation anomaly detection. For example, in products such as photography, video editing, short videos, and live streaming, segmentation models are often used to locate human figures to achieve video effects such as background replacement, sky replacement, and beauty filters. In beauty effect applications, the segmentation model can achieve anti-occlusion functionality, meaning that when occlusions cover facial features such as lips, eyes, and eyebrows, the makeup effect will not be displayed above the occluded area. For use cases with poor segmentation results, it is necessary to detect whether the image segmentation results are abnormal and use images with abnormal segmentation results to supplement the dataset to optimize the segmentation model. Existing image segmentation anomaly detection schemes typically rely on manual screening, where staff observe the image segmentation results and determine if they are abnormal. However, this method is inefficient. Therefore, this application provides an image segmentation anomaly detection method to address the inefficiency of existing schemes.

[0027] Figure 1 A flowchart of an image segmentation anomaly detection method provided in this application embodiment is given. The image segmentation anomaly detection method provided in this application embodiment can be executed by an image segmentation anomaly detection device, which can be implemented by hardware and / or software and integrated into an image segmentation anomaly detection equipment.

[0028] The following description uses an image segmentation anomaly detection device to perform an image segmentation anomaly detection method as an example.

[0029] refer to Figure 1 The image segmentation anomaly detection method includes:

[0030] S110: Obtain the current target bounding box image, the previous target bounding box image, the current segmentation output mask, and the previous segmentation output mask.

[0031] For example, the current target bounding box image, the previous target bounding box image, the current segmentation output mask, and the previous segmentation output mask are obtained. Specifically, the current target bounding box image and the current segmentation output mask provided in this solution are obtained by segmenting the current input image using a predefined image segmentation model (e.g., face segmentation model, hair segmentation model, facial feature segmentation model, sky segmentation model, car segmentation model, etc.). The previous target bounding box image and the previous segmentation output mask are obtained by segmenting the previous input image using the same image segmentation model. The previous input image and the current input image are two frames that are sequentially input into the image segmentation model for image segmentation.

[0032] For example, when using an image segmentation model to segment faces in video data (complete video or video stream), the model sequentially performs image segmentation processing on each frame of the video data (i.e., the input image), determining the target bounding box (e.g., the face bounding box) and the corresponding image segmentation result (i.e., the mask image) in the image. After each frame of the input image is segmented, the face bounding box image and image segmentation result corresponding to that frame of the input image, as well as the face bounding box image and image segmentation result corresponding to the input image segmented in the previous frame, are obtained and used as the current target bounding box image and the current segmentation output mask, and the previous target bounding box image and the previous segmentation output mask, respectively.

[0033] In one embodiment, when segmenting an input image using a face segmentation model, the face segmentation model determines the bounding box corresponding to the face region in the input image, multiple facial key points within the bounding box, the face confidence score corresponding to the bounding box, and the mask. The mask image determines the regions corresponding to the face and occluders (e.g., the face region is white, and the background and occluder regions are black). The probability value of each pixel in the mask ranges from [0, 255], with a higher probability value indicating a greater probability that the pixel belongs to the face region.

[0034] S120: Determine the spatial anomaly probability of the current segmentation output mask in the spatial domain based on the percentage of pixels whose probability values ​​are within a set probability range in the current segmentation output mask.

[0035] For example, the proportion of pixels in the current segmentation output mask whose probability values ​​fall within a set probability range is determined, and the spatial anomaly probability of the current segmentation output mask in the spatial domain is determined based on this proportion. The larger the proportion of pixels in the current segmentation output mask whose probability values ​​fall within the set probability range, the higher the corresponding spatial anomaly probability. A higher spatial anomaly probability indicates a greater likelihood that the current segmentation output mask is abnormal.

[0036] It should be explained that image segmentation models output probability values ​​closer to 0 or 255 for simple samples (relatively easy inputs), while for difficult samples (inputs that are not easily segmented correctly), the output probability value is at a moderate level (closer to 127.5). Based on this, the spatial anomaly probability of the mask in the spatial domain can be calculated by detecting the proportion of pixels with moderate probability in the mask output by the image segmentation model.

[0037] Optionally, the set probability range can be the range corresponding to the medium probability. For example, when the value range of each pixel of the mask is [0, 255], the corresponding medium probability is 127.5. The range corresponding to the value distance before and after 127.5 can be used as the set probability range. For example, the set probability range can be [64, 192], [10, 240], [131, 155], etc.

[0038] S130: Based on the color value difference between the current target bounding box image and the previous target bounding box image, and the probability value difference between the current segmentation output mask and the previous segmentation output mask, determine the temporal anomaly probability of the current segmentation output mask in the temporal domain.

[0039] For example, the color value difference between the current target bounding box image and the previous target bounding box image, and the probability value difference between the current segmentation output mask and the previous segmentation output mask are determined. The temporal anomaly probability of the current segmentation output mask is then determined based on the color value difference and the probability value difference. The larger the color value difference and the probability value difference, the higher the corresponding temporal anomaly probability. A higher temporal anomaly probability indicates a greater likelihood of anomalies in the current segmentation output mask. Optionally, after determining the temporal anomaly probability, an exponential function (e.g., y = x) can be further utilized. 2 The temporal anomaly probability is proportionally transformed into a probability value. In one embodiment, the color value difference can be represented by the difference in color values ​​between the current target bounding box image and the previous target bounding box image, and the probability value difference can be represented by the difference in probability values ​​between the current segmentation output mask and the previous segmentation output mask. The color value difference and probability value difference can also be represented by the difference in the absolute value of the color value and the probability value, the difference in the image quality index (PSNR), etc., respectively.

[0040] It should be explained that when the bounding box image in the input image of the image segmentation model does not change much compared to the bounding box image in the previous frame, the output result of the image segmentation model should also not change much compared to the output result of the previous frame. Based on this, the temporal anomaly probability of the mask can be calculated in the temporal domain according to the difference in color values ​​of the bounding box images before and after, as well as the difference in probability values ​​of the output masks before and after.

[0041] S140: Determine the anomaly detection result of the current segmentation output mask based on the spatial anomaly probability and the temporal anomaly probability.

[0042] For example, after determining the spatial anomaly probability and the temporal anomaly probability, the anomaly detection result of the current segmentation output mask is determined based on the spatial and temporal anomaly probabilities. For instance, the maximum value between the spatial and temporal anomaly probabilities can be used as the anomaly detection result of the current segmentation output mask; alternatively, the sum of the spatial and temporal anomaly probabilities can be used as the anomaly detection result of the current segmentation output mask; or, the spatial and temporal anomaly probabilities can be weighted and summed, and the weighted sum (truncated to [0,1]) can be used as the anomaly detection result of the current segmentation output mask.

[0043] In one embodiment, after determining the anomaly detection result of the current segmentation output mask, if the anomaly detection result of the current segmentation output mask indicates an image segmentation result anomaly, one or more combinations of the current segmentation output mask, the current target bounding box image, and the current input image can be added to the sample dataset for optimized training of the image segmentation model, thereby improving the image segmentation accuracy of the image segmentation model. For example, during the operation of the image segmentation model (such as during live streaming, when the broadcaster uses a makeup effect based on the image segmentation result of the image segmentation model), scenarios with image segmentation anomalies can be detected in real time, and image segmentation anomalies can be reported to the server for image segmentation anomaly data collection. The server periodically adds the collected data, which can be manually or machine-annotated, to the training dataset of the segmentation model, retraining the segmentation model. The retrained segmentation model can overcome the performance of the previous version of the image segmentation model in scenarios with supplemented data, thereby providing users with better image segmentation processing results.

[0044] The above describes a method that determines the spatial anomaly probability of the current segmentation output mask in the spatial domain by considering the percentage of pixels with probability values ​​within a set probability range in the current segmentation output mask. It also determines the temporal anomaly probability of the current segmentation output mask in the temporal domain by considering the color value difference between the current target bounding box image and the previous target bounding box image, as well as the probability value difference between the current segmentation output mask and the previous segmentation output mask. Based on these spatial and temporal anomaly probabilities, the method determines the anomaly detection result of the current segmentation output mask. This allows for the evaluation of the degree of anomaly in the image segmentation results in both the spatial and temporal domains, accurately determining whether the image segmentation results are abnormal. This eliminates the need for manual screening of abnormal segmentation results and effectively improves the efficiency of image segmentation anomaly detection.

[0045] Based on the above embodiments, Figure 2 A flowchart of another image segmentation anomaly detection method provided in this application embodiment is given, which is a concretization of the above-described image segmentation anomaly detection method. (Reference) Figure 2 The image segmentation anomaly detection method includes:

[0046] S210: Obtain the current target bounding box image, the previous target bounding box image, the current segmentation output mask, and the previous segmentation output mask.

[0047] The current target bounding box image and the current segmentation output mask are obtained by segmenting the current input image using a set image segmentation model, while the previous target bounding box image and the previous segmentation output mask are obtained by segmenting the previous input image using an image segmentation model.

[0048] For example, the current target bounding box image, the previous target bounding box image, the current segmentation output mask, and the previous segmentation output mask are obtained, wherein the current target bounding box image, the previous target bounding box image, the current segmentation output mask, and the previous segmentation output mask have the same size.

[0049] For example, the current target bounding box image I t The width and height are 112x112, and the current target bounding box image I t The image is a 3-channel RGB image, with each pixel ranging from [0, 255]; the current segmentation output mask M t The width and height are 112x112, and the current segmentation output mask M t This is a single-channel grayscale image, where the value of each pixel ranges from [0, 255]; the previous target bounding box image I t-1 The width and height are 112x112, and the previous target box image I t-1 The image is a 3-channel RGB image, with each pixel ranging from [0, 255]; the previous segmentation output mask M t-1 The width and height are 112x112, and the previous segmented output mask M t-1It is a single-channel grayscale image, and the value of each pixel is in the range of [0, 255].

[0050] S220: Determine the number of low-confidence pixels in the current segmentation output mask whose probability values ​​are within a set probability range, and the number of valid rows in the current segmentation output mask containing pixels whose probability values ​​are within a set probability range.

[0051] For example, the number of low-confidence pixels in the current segmentation output mask whose probability values ​​fall within a set probability range is determined. Here, low-confidence pixels are pixels whose execution degree falls within a medium probability range, and the set probability range is the range corresponding to the medium probability. Optionally, the set probability range can be [64, 192], that is, determining the number of low-confidence pixels in the current segmentation output mask whose probability values ​​fall within [64, 192]. Here, [64, 192] in the [0, 255] range is approximately [0.25, 0.75] when converted to the [0, 1] interval.

[0052] Furthermore, the method identifies valid rows in the current segmentation output mask containing pixels with probability values ​​within a set probability range, and determines the number of valid rows. This approach determines the validity of rows in the current segmentation output mask, ensuring they are meaningful for anomaly detection in image segmentation. This reduces the waste of computational resources caused by rows with no faces in the current segmentation output mask participating in anomaly detection due to face angle issues, thus improving the efficiency of anomaly detection in image segmentation.

[0053] In one embodiment, the number of low-confidence pixels with probability values ​​within a set probability range in the current segmentation output mask, and the number of valid rows containing pixels with probability values ​​within the set probability range, can be determined according to a set scanning interval. The set scanning interval includes a row scanning interval and a column scanning interval. For example, if the set scanning interval is set to scan once every 7 rows, a total of 112 / 7 = 16 rows are scanned. For each row, every 4 columns are scanned once, a total of 112 / 4 = 28 columns are scanned. For a current segmentation output mask with a width and height of 112x112, a total of 112 / 7 * 112 / 4 = 448 pixels are scanned. By determining the number of low-confidence pixels and the number of valid rows according to the set scanning interval, the amount of data computation is reduced, effectively improving the speed of image segmentation anomaly detection while ensuring the accuracy of image segmentation anomaly detection.

[0054] S230: Determine the number of normal low-confidence pixels and the number of abnormal low-confidence pixels based on the number of valid rows.

[0055] For example, the number of normal low-confidence pixels and the number of abnormal low-confidence pixels are determined based on the aforementioned number of valid rows. This scheme uses the product of a set first coefficient and the number of valid rows as the number of normal low-confidence pixels, and the product of a set second coefficient and the number of valid rows as the number of abnormal low-confidence pixels. Both the first and second coefficients are greater than 1, and the second coefficient is greater than the first coefficient. For normal face segmentation results, since there is a probability value jump from 0 to 255 at the face boundary, there is generally one low-confidence pixel. The first coefficient can be set to 2, and the second coefficient can be set to 6. Then, the product of the number of valid rows and 2 is used as the number of normal low-confidence pixels, and the product of the number of valid rows and 6 is used as the number of abnormal low-confidence pixels.

[0056] S240: The number of low-confidence pixels is linearly normalized based on the number of normal low-confidence pixels and the number of abnormal low-confidence pixels to obtain the spatial domain anomaly probability of the current segmentation output mask in the spatial domain.

[0057] For example, the number of low-confidence pixels is subjected to min-max linear normalization based on the number of normal low-confidence pixels and the number of abnormal low-confidence pixels, normalizing the number of low-confidence pixels to the range [0, 1], and the result of the linear normalization is used as the spatial anomaly probability of the current segmentation output mask in the spatial domain. For example, the spatial anomaly probability can be determined based on the following linear normalization formula:

[0058] P s =(L cca -L CCN ) / (L CCM -L CCN )

[0059] Among them, P s L represents the spatial anomaly probability. cca For the number of low-confidence pixels, L CCN For the normal number of low-confidence pixels, L CCM This represents the number of pixels with abnormally low confidence.

[0060] This scheme accurately determines the spatial anomaly probability of the current segmentation output mask in the spatial domain by linearly normalizing the number of normal low-confidence pixels and abnormal low-confidence pixels in the current segmentation output mask, and accurately evaluates the degree of anomaly of the image segmentation result in the spatial domain, thus effectively improving the accuracy of image segmentation anomaly detection.

[0061] In one possible embodiment, the image segmentation anomaly detection method provided by this solution, after determining the spatial anomaly probability of the current segmentation output mask in the spatial domain based on the proportion of pixels with probability values ​​within a set probability range in the current segmentation output mask, can also use a set exponential function to transform the spatial anomaly probability, wherein the exponent of the set exponential function is less than 1.

[0062] For example, after determining the spatial anomaly probability of the current segmentation output mask in the spatial domain, the spatial anomaly probability is transformed using a defined exponential function. For instance, the spatial anomaly probability can be transformed using the following formula:

[0063] P S =P′ S ^a

[0064] Among them, P S Let P′ be the spatial anomaly probability after transformation. S Let be the spatial anomaly probability before transformation, and 'a' be a predefined exponential function, for example, a = 0.5. This scheme transforms the spatial anomaly probability by setting an exponential function, making the spatial anomaly probability more sensitive to minor anomalies, thereby improving the sensitivity and accuracy of anomaly detection in image segmentation.

[0065] S250: Determine the proportion of first pixels whose color value difference at the same position between the current target bounding box image and the previous target bounding box image reaches the first difference threshold.

[0066] S260: Determine the proportion of the second pixel where the probability difference between the current segmentation output mask and the previous segmentation output mask at the same position reaches the second difference threshold.

[0067] For example, the pixels whose color values ​​differ by a first difference threshold at the same location between the current target bounding box image and the previous target bounding box image are calculated, and the proportion of these pixels in the current target bounding box image is determined. Optionally, the first difference threshold is 64, that is, the pixels whose color values ​​differ by a first difference threshold of 64 at the same location between the current target bounding box image and the previous target bounding box image are calculated, and the proportion of these pixels whose color values ​​differ by a first difference threshold of 64 in the current target bounding box image is determined.

[0068] Furthermore, the probability values ​​of pixels at the same position in the current segmentation output mask and the previous segmentation output mask that differ by a second difference threshold are calculated, and the proportion of these pixels in the second pixel of the current segmentation output mask is determined. Optionally, the second difference threshold is 64, that is, the probability values ​​of pixels at the same position in the current segmentation output mask and the previous segmentation output mask that differ by a probability value of 64 are calculated, and the proportion of these pixels with a probability value difference of 64 in the second pixel of the current segmentation output mask is determined.

[0069] S270: Determine the jitter level of the segmentation result based on the ratio of the first pixel and the ratio of the second pixel.

[0070] For example, the jitter level of the segmentation result corresponding to the current segmentation output mask is determined based on the first pixel ratio and the second pixel ratio determined above. Optionally, the jitter level of the segmentation result can be determined based on the quotient of the second pixel ratio and the first pixel ratio.

[0071] In one possible embodiment, the image segmentation anomaly detection method provided by this solution, when determining the jitter level of the segmentation result based on the first pixel ratio and the second pixel ratio, includes at least one of the following methods:

[0072] Method 1: Determine the jitter level of the segmentation result based on the ratio of the second pixel ratio to the first pixel ratio.

[0073] Method 2: Determine the jitter level of the segmentation result based on the ratio of the second pixel ratio and the ratio of the first pixel ratio to the maximum value in the set adjustment parameters.

[0074] For example, in method one, the ratio of the second pixel ratio to the first pixel ratio is calculated, and this ratio is determined as the jitter level of the segmentation result. For instance, the jitter level of the segmentation result can be expressed as: R f =D m / max(b,D i ), where D m The second pixel ratio is 'b', which is the adjustment parameter, and 'D' is the value of the second pixel. i The ratio of the first pixel to the current target bounding box image is used. This scheme determines the jitter level of the segmentation result by comparing the ratio of the second pixel ratio to the first pixel ratio, thereby improving the efficiency of anomaly detection in image segmentation.

[0075] For Method 2, the first pixel ratio is determined and the maximum value of the set adjustment parameters is set. The ratio of the second pixel ratio to the determined maximum value is used as the jitter level of the segmentation result. Optionally, the set adjustment parameter can be 0.01. When the input image is close to a static image, the ratio of the second pixel ratio to the first pixel ratio is prone to instability and becomes very large. By setting the adjustment parameter as the minimum value of the second pixel ratio, the occurrence of this instability can be reduced, and the accuracy of image segmentation anomaly detection can be improved.

[0076] S280: Map the jitter of the segmentation result to a set probability range to obtain the temporal anomaly probability of the current segmentation output mask in the time domain.

[0077] For example, the determined jitter level of the segmentation result is mapped to a set probability range, and the mapping result is used as the temporal anomaly probability of the current segmentation output mask in the time domain. Optionally, the set probability range can be [0,1]. This scheme determines the jitter level of the segmentation result based on the first pixel ratio on the target bounding box image and the second pixel ratio on the segmentation output mask, and maps the jitter level of the segmentation result to a set probability range, accurately determining the temporal anomaly probability of the current segmentation output mask in the time domain, accurately evaluating the anomaly level of the image segmentation result in the time domain, and effectively improving the accuracy of image segmentation anomaly detection.

[0078] In one possible embodiment, the image segmentation anomaly detection method provided by this solution can obtain the temporal anomaly probability of the current segmentation output mask in the time domain by mapping the jitter level of the segmentation result to a set probability range based on a set jitter level threshold and a set probability threshold, by linearly scaling the jitter level of the segmentation result to a set probability range.

[0079] In one embodiment, the minimum of the ratio of the segmentation result jitter to a set jitter threshold and a set probability threshold can be used as the scaling result of linearly scaling the segmentation result jitter to a set probability range. This scaling result is the temporal anomaly probability of the current segmentation output mask in the time domain. For example, the segmentation result jitter can be linearly scaled to a set probability range using the following formula:

[0080] R nf =min(R) f / R f_max c)

[0081] Among them, R nf For scaling results, R f R represents the degree of jitter in the segmentation results. f_max R is the threshold for jitter level, c is the set probability threshold, which is optional. f_matx =3, c=1. When the jitter level of the segmentation result reaches the jitter level threshold, it means that the jitter level of the segmentation result is a multiple of the jitter level threshold of the segmented input image, which can be considered as a relatively serious segmentation anomaly. At this time, the temporal anomaly probability is the set probability threshold. This scheme accurately determines the temporal anomaly probability by linearly scaling the jitter level of the segmentation result to the set probability range based on the jitter level threshold and the set probability threshold, thereby improving the accuracy of image segmentation anomaly detection.

[0082] In one possible embodiment, the image segmentation anomaly detection method provided by this solution, after mapping the jitter of the segmentation result to a set probability range to obtain the temporal anomaly probability of the current segmentation output mask in the time domain, can also determine the static degree of the current target box image and perform weighted processing on the temporal anomaly probability according to the static degree.

[0083] For example, the static level of the current target bounding box image is determined, and the temporal anomaly probability (i.e., the scaling result determined above) is weighted according to the static level. The weighted result is then used as the final temporal anomaly probability. Optionally, the temporal anomaly probability can be weighted using the following formula:

[0084] P t =R nf *S

[0085] Among them, P t R represents the weighted time-domain anomaly probability. nf The temporal anomaly probability before weighted processing (i.e., the scaling result determined above) is denoted by S, which represents the static level of the current target bounding box image. This scheme obtains the final temporal anomaly probability by weighting the temporal anomaly probability according to the static level of the current target bounding box image. This ensures that even if the segmentation model performs slightly worse when the image changes drastically, it will not significantly affect the live broadcast performance, while maintaining the image segmentation anomaly detection effect.

[0086] In one embodiment, the image segmentation anomaly detection method provided by this solution can determine the static level of the current target bounding box image based on color value differences. Optionally, the static level of the current target bounding box image can be determined based on the following formula.

[0087] S = 2^(-20*Di)

[0088] Among them, D i S represents the first pixel ratio corresponding to the current target bounding box image. When the input image is completely static (Di = 0), S = 1, which has no effect on Rnf. When the input image changes by 5% (Di = 0.05), S = 0.5, and Rnf... nf It will be reduced to half of its original value.

[0089] S290: Determine the anomaly detection result of the current segmentation output mask based on the spatial anomaly probability and the temporal anomaly probability.

[0090] The above describes a method that determines the spatial anomaly probability of the current segmentation output mask in the spatial domain by calculating the proportion of pixels with probability values ​​within a set probability range in the current segmentation output mask. It also determines the temporal anomaly probability of the current segmentation output mask in the temporal domain by calculating the color value difference between the current target bounding box image and the previous target bounding box image, as well as the probability value difference between the current and previous segmentation output masks. Based on these spatial and temporal anomaly probabilities, the method determines the anomaly detection result of the current segmentation output mask. This allows for the evaluation of the degree of anomaly in the image segmentation results in both the spatial and temporal domains, accurately determining whether the image segmentation results are abnormal without requiring manual screening of abnormal segmentation results, thus effectively improving the efficiency of image segmentation anomaly detection. Furthermore, by linearly normalizing the number of low-confidence pixels based on the number of normal and abnormal low-confidence pixels in the current segmentation output mask, the method accurately determines the spatial anomaly probability of the current segmentation output mask in the spatial domain, accurately evaluating the degree of anomaly in the image segmentation results in the spatial domain, and effectively improving the accuracy of image segmentation anomaly detection. Furthermore, by weighting the temporal anomaly probability according to the staticity of the current target bounding box image, the final temporal anomaly probability is obtained. This ensures that even if the segmentation model performs slightly worse when the image changes drastically, it will not significantly affect the live broadcast performance, while maintaining the image segmentation anomaly detection effect.

[0091] Figure 3 This is a schematic diagram of the structure of an image segmentation anomaly detection device provided in an embodiment of this application. (Reference) Figure 3 The image segmentation anomaly detection device includes an image acquisition module 31, a spatial domain detection module 32, a temporal domain detection module 33, and an anomaly determination module 34.

[0092] The image acquisition module 31 is configured to acquire the current target box image, the previous target box image, the current segmentation output mask, and the previous segmentation output mask. The current target box image and the current segmentation output mask are obtained by performing image segmentation on the current input image through a set image segmentation model. The previous target box image and the previous segmentation output mask are obtained by performing image segmentation on the previous input image through an image segmentation model.

[0093] The spatial domain detection module 32 is configured to determine the spatial domain anomaly probability of the current segmentation output mask in the spatial domain based on the proportion of pixels whose probability values ​​are within a set probability range in the current segmentation output mask.

[0094] The temporal detection module 33 is configured to determine the temporal anomaly probability of the current segmentation output mask in the temporal domain based on the color value difference between the current target box image and the previous target box image, and the probability value difference between the current segmentation output mask and the previous segmentation output mask.

[0095] The anomaly determination module 34 is configured to determine the anomaly detection result of the current segmentation output mask based on the spatial anomaly probability and the temporal anomaly probability.

[0096] The above describes a method that determines the spatial anomaly probability of the current segmentation output mask in the spatial domain by considering the percentage of pixels with probability values ​​within a set probability range in the current segmentation output mask. It also determines the temporal anomaly probability of the current segmentation output mask in the temporal domain by considering the color value difference between the current target bounding box image and the previous target bounding box image, as well as the probability value difference between the current segmentation output mask and the previous segmentation output mask. Based on these spatial and temporal anomaly probabilities, the method determines the anomaly detection result of the current segmentation output mask. This allows for the evaluation of the degree of anomaly in the image segmentation results in both the spatial and temporal domains, accurately determining whether the image segmentation results are abnormal. This eliminates the need for manual screening of abnormal segmentation results and effectively improves the efficiency of image segmentation anomaly detection.

[0097] In one possible embodiment, when the spatial domain detection module 32 determines the spatial domain anomaly probability of the current segmentation output mask in the spatial domain based on the proportion of pixels in the current segmentation output mask whose probability values ​​are within a set probability range, it is configured as follows:

[0098] Determine the number of low-confidence pixels in the current segmentation output mask whose probability values ​​are within a set probability range, and the number of valid rows in the current segmentation output mask containing pixels whose probability values ​​are within a set probability range;

[0099] The number of normal low-confidence pixels and the number of abnormal low-confidence pixels are determined based on the number of valid rows;

[0100] The number of low-confidence pixels is linearly normalized based on the number of normal low-confidence pixels and the number of abnormal low-confidence pixels to obtain the spatial anomaly probability of the current segmentation output mask in the spatial domain.

[0101] In one possible embodiment, the airspace detection module 32 is further configured to transform the airspace anomaly probability using a set exponential function, wherein the exponent of the set exponential function is less than 1.

[0102] In one possible embodiment, when the temporal detection module 33 determines the temporal anomaly probability of the current segmentation output mask in the temporal domain based on the color value difference between the current target bounding box image and the previous target bounding box image, and the probability value difference between the current segmentation output mask and the previous segmentation output mask, it is configured as follows:

[0103] Determine the proportion of first pixels whose color value difference at the same location between the current target bounding box image and the previous target bounding box image reaches a first difference threshold.

[0104] Determine the proportion of the second pixel where the probability value difference between the current segmentation output mask and the previous segmentation output mask at the same position reaches the second difference threshold;

[0105] The degree of jitter in the segmentation result is determined based on the ratio of the first pixel and the ratio of the second pixel.

[0106] By mapping the jitter of the segmentation result to a set probability range, the temporal anomaly probability of the current segmentation output mask in the time domain is obtained.

[0107] In one possible embodiment, when determining the jitter level of the segmentation result based on the first pixel ratio and the second pixel ratio, the temporal detection module 33 is configured to be at least one of the following:

[0108] The degree of jitter in the segmentation result is determined based on the ratio of the second pixel ratio to the first pixel ratio;

[0109] The degree of jitter in the segmentation result is determined by the ratio of the second pixel ratio to the first pixel ratio and the maximum value in the set adjustment parameters.

[0110] In one possible embodiment, when the temporal detection module 33 maps the jitter level of the segmentation result to a set probability range to obtain the temporal anomaly probability of the current segmentation output mask in the temporal domain, it includes:

[0111] Based on the set jitter threshold and the set probability threshold, the jitter of the segmentation result is linearly scaled to the set probability range to obtain the temporal anomaly probability of the current segmentation output mask in the time domain.

[0112] In one possible embodiment, the temporal detection module 33 is further configured to determine the static degree of the current target bounding box image and to weight the temporal anomaly probability according to the static degree.

[0113] In one possible embodiment, the temporal detection module 33 is configured to determine the static level of the current target bounding box image based on the color value difference when determining the static level of the current target bounding box image.

[0114] It is worth noting that in the embodiments of the above-mentioned image segmentation anomaly detection device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.

[0115] This application also provides an image segmentation anomaly detection device, which can integrate the image segmentation anomaly detection apparatus provided in this application. Figure 4 This is a schematic diagram of the structure of an image segmentation anomaly detection device provided in an embodiment of this application. (Reference) Figure 4The image segmentation anomaly detection device includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when one or more programs are executed by one or more processors 41, the one or more processors 41 implement the image segmentation anomaly detection method provided in the above embodiments. The image segmentation anomaly detection device, apparatus, and computer provided above can be used to execute the image segmentation anomaly detection method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0116] This application also provides a non-volatile storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform the image segmentation anomaly detection method provided in the above embodiments. Of course, the computer-executable instructions provided in this application are not limited to the image segmentation anomaly detection method provided above; they can also perform related operations in the image segmentation anomaly detection method provided in any embodiment of this application. The image segmentation anomaly detection apparatus, device, and storage medium provided in the above embodiments can execute the image segmentation anomaly detection method provided in any embodiment of this application. Technical details not described in detail in the above embodiments can be found in the image segmentation anomaly detection method provided in any embodiment of this application.

[0117] Based on the above embodiments, this application also provides a computer program product. The technical solution of this application, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes several instructions to cause a computer device, mobile terminal, or processor therein to execute all or part of the steps of the image segmentation anomaly detection method provided in the various embodiments of this application.

Claims

1. An image segmentation anomaly detection method, characterized in that, include: The current target bounding box image, the previous target bounding box image, the current segmentation output mask, and the previous segmentation output mask are obtained. The current target bounding box image and the current segmentation output mask are obtained by performing image segmentation on the current input image using a set image segmentation model. The previous target bounding box image and the previous segmentation output mask are obtained by performing image segmentation on the previous input image using the image segmentation model. Based on the percentage of pixels whose probability values ​​are within a set probability range in the current segmentation output mask, the spatial anomaly probability of the current segmentation output mask in the spatial domain is determined. Based on the color value difference between the current target bounding box image and the previous target bounding box image, and the probability value difference between the current segmentation output mask and the previous segmentation output mask, the temporal anomaly probability of the current segmentation output mask in the time domain is determined. The anomaly detection result of the current segmentation output mask is determined based on the spatial anomaly probability and the temporal anomaly probability.

2. The image segmentation anomaly detection method according to claim 1, characterized in that, The step of determining the spatial anomaly probability of the current segmentation output mask in the spatial domain based on the proportion of pixels whose probability values ​​are within a set probability range includes: Determine the number of low-confidence pixels in the current segmentation output mask whose probability values ​​are within a set probability range, and the number of valid rows in the current segmentation output mask containing pixels whose probability values ​​are within the set probability range; The number of normal low-confidence pixels and the number of abnormal low-confidence pixels are determined based on the number of valid rows. The number of low-confidence pixels is linearly normalized based on the number of normal low-confidence pixels and the number of abnormal low-confidence pixels to obtain the spatial anomaly probability of the current segmentation output mask in the spatial domain.

3. The image segmentation anomaly detection method according to claim 1, characterized in that, After determining the spatial anomaly probability of the current segmentation output mask in the spatial domain based on the proportion of pixels with probability values ​​within a set probability range in the current segmentation output mask, the method further includes: The spatial anomaly probability is transformed using a predefined exponential function, wherein the exponent of the predefined exponential function is less than 1.

4. The image segmentation anomaly detection method according to claim 1, characterized in that, The step of determining the temporal anomaly probability of the current segmentation output mask in the temporal domain based on the color value difference between the current target bounding box image and the previous target bounding box image, and the probability value difference between the current segmentation output mask and the previous segmentation output mask, includes: Determine the first pixel ratio at which the color value difference between the current target bounding box image and the previous target bounding box image at the same position reaches a first difference threshold; Determine the second pixel ratio at which the probability value difference between the current segmentation output mask and the previous segmentation output mask at the same position reaches a second difference threshold; The degree of jitter in the segmentation result is determined based on the first pixel ratio and the second pixel ratio; The jitter level of the segmentation result is mapped to a set probability range to obtain the temporal anomaly probability of the current segmentation output mask in the time domain.

5. The image segmentation anomaly detection method according to claim 4, characterized in that, Determining the jitter level of the segmentation result based on the first pixel ratio and the second pixel ratio includes at least one of the following: The degree of jitter in the segmentation result is determined based on the ratio of the second pixel ratio to the first pixel ratio; The degree of jitter in the segmentation result is determined based on the ratio of the second pixel ratio and the first pixel ratio to the maximum value in the set adjustment parameters.

6. The image segmentation anomaly detection method according to claim 4, characterized in that, The step of mapping the jitter level of the segmentation result to a set probability range to obtain the temporal anomaly probability of the current segmentation output mask in the time domain includes: Based on the set jitter threshold and the set probability threshold, the jitter of the segmentation result is linearly scaled to the set probability range to obtain the temporal anomaly probability of the current segmentation output mask in the time domain.

7. The image segmentation anomaly detection method according to claim 4, characterized in that, After mapping the jitter level of the segmentation result to a set probability range to obtain the temporal anomaly probability of the current segmentation output mask in the time domain, the method further includes: The static degree of the current target bounding box image is determined, and the temporal anomaly probability is weighted according to the static degree.

8. The image segmentation anomaly detection method according to claim 7, characterized in that, Determining the static degree of the current target bounding box image includes: The static level of the current target bounding box image is determined based on the color value difference.

9. An image segmentation anomaly detection device, characterized in that, It includes an image acquisition module, a spatial domain detection module, a temporal domain detection module, and an anomaly determination module, wherein: The image acquisition module is configured to acquire the current target bounding box image, the previous target bounding box image, the current segmentation output mask, and the previous segmentation output mask. The current target bounding box image and the current segmentation output mask are obtained by performing image segmentation on the current input image using a set image segmentation model. The previous target bounding box image and the previous segmentation output mask are obtained by performing image segmentation on the previous input image using the image segmentation model. The spatial domain detection module is configured to determine the spatial domain anomaly probability of the current segmentation output mask in the spatial domain based on the percentage of pixels whose probability values ​​are within a set probability range in the current segmentation output mask. The temporal detection module is configured to determine the temporal anomaly probability of the current segmentation output mask in the temporal domain based on the color value difference between the current target bounding box image and the previous target bounding box image, and the probability value difference between the current segmentation output mask and the previous segmentation output mask. The anomaly determination module is configured to determine the anomaly detection result of the current segmentation output mask based on the spatial anomaly probability and the temporal anomaly probability.

10. An image segmentation anomaly detection device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image segmentation anomaly detection method as described in any one of claims 1-8.

11. A non-volatile storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the image segmentation anomaly detection method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image segmentation anomaly detection method according to any one of claims 1-8.

Images