Image anomaly detection method and device, computer device and storage medium

Abstract

The application discloses an image anomaly detection method and device, computer equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: determining a plurality of superpixel regions of an image, wherein the superpixel regions comprise a plurality of pixel points in the image; extracting k first superpixel features of different scales from each superpixel region, wherein k is an integer greater than 1; fusing the k first superpixel features of different scales of each superpixel region to obtain a superpixel fusion feature of each superpixel region; and determining an anomaly detection result of each superpixel region based on the superpixel fusion feature of each superpixel region, wherein the anomaly detection result represents a probability that the superpixel region in the image is abnormal. The application extracts and fuses local features in units of superpixel regions to identify the probability that each superpixel region is abnormal, can reduce unnecessary global operation, reduces the operation complexity, and improves the efficiency of anomaly detection.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image anomaly detection method, apparatus, computer device, and storage medium. Background Technology

[0002] Devices may exhibit abnormal phenomena when displaying images, such as color distortion, blurry content, content misalignment, flickering, stripes, or garbled text. Therefore, detecting these abnormal phenomena in videos, games, or live streams is a pressing issue that needs to be addressed.

[0003] In related technologies, the detection of anomalies in display images mainly relies on manual review, which is labor-intensive and time-consuming, resulting in low efficiency. Summary of the Invention

[0004] This application provides an image anomaly detection method, apparatus, computer device, and storage medium, which can improve the efficiency of anomaly detection. The technical solution is as follows:

[0005] On the one hand, an image anomaly detection method is provided, the method comprising:

[0006] Identify multiple superpixel regions of an image, wherein the superpixel regions include multiple pixels in the image;

[0007] Extract k first superpixel features of different scales from each superpixel region, where k is an integer greater than 1;

[0008] By fusing the k different scales of the first superpixel features of each superpixel region, a superpixel fused feature for each superpixel region is obtained;

[0009] Based on the superpixel fusion features of each superpixel region, an anomaly detection result is determined for each superpixel region, wherein the anomaly detection result represents the probability that an anomaly exists in the superpixel region of the image.

[0010] On the other hand, an image anomaly detection device is provided, the device comprising:

[0011] A determination module is used to determine multiple superpixel regions of an image, wherein the superpixel regions include multiple pixels in the image;

[0012] The feature extraction module is used to extract k first superpixel features of different scales from each superpixel region, where k is an integer greater than 1;

[0013] The feature fusion module is used to fuse the k different scales of the first superpixel features of each superpixel region to obtain the superpixel fused feature of each superpixel region.

[0014] The detection module is used to determine the anomaly detection result of each superpixel region based on the superpixel fusion feature of each superpixel region, wherein the anomaly detection result represents the probability that there is an anomaly in the superpixel region of the image.

[0015] Optionally, the determining module is configured to:

[0016] Multiple cluster center points were identified in the image;

[0017] Based on multiple clustering center points, multiple pixels in the image are clustered to obtain multiple clusters;

[0018] The region formed by multiple pixels in any of the clusters is defined as a superpixel region.

[0019] Optionally, the determining module is configured to:

[0020] For any given pixel, the feature distance between the pixel and the multiple cluster centers is determined based on the color distance and spatial distance between the pixel and the multiple cluster centers, respectively. The color distance refers to the distance between the color parameters of the pixel and the color parameters of the cluster centers, and the spatial distance refers to the distance between the position parameters of the pixel and the position parameters of the cluster centers.

[0021] The pixel is assigned to the cluster containing the cluster center point with the smallest feature distance to the pixel.

[0022] Optionally, the anomaly detection model includes k feature extraction networks of different scales, each of which includes multiple feature extraction layers connected in sequence;

[0023] The feature extraction module is used for:

[0024] The superpixel regions are respectively input into k feature extraction networks in the anomaly detection model to obtain the superpixel features output by the first feature extraction layer and the superpixel features output by the last feature extraction layer in the k feature extraction networks;

[0025] For each of the feature extraction networks, the superpixel features output by the first feature extraction layer and the superpixel features output by the last feature extraction layer are fused to obtain k first superpixel features of different scales.

[0026] Optionally, the feature fusion module is used to:

[0027] By fusing k first superpixel features of different scales in the superpixel region, a second superpixel feature of the superpixel region is obtained;

[0028] Determine the adjacent superpixel regions of the superpixel region, where the adjacent superpixel regions refer to other superpixel regions adjacent to the superpixel region;

[0029] The second superpixel feature of the superpixel region and the second superpixel feature of the adjacent superpixel region are fused to obtain the superpixel fused pixel feature of the superpixel region.

[0030] Optionally, the feature fusion module is used to:

[0031] Based on the difference between the second superpixel feature of the superpixel region and the second superpixel feature of the adjacent superpixel region, a weight is determined between the superpixel region and the adjacent superpixel region, and the weight is negatively correlated with the difference.

[0032] Based on the weights between the superpixel region and the adjacent superpixel region, the second superpixel features of the adjacent superpixel region are weighted to obtain the third superpixel features of the adjacent superpixel region.

[0033] The second superpixel feature of the superpixel region is added to the third superpixel feature of the adjacent superpixel region to obtain the superpixel fusion feature of the superpixel region.

[0034] Optionally, the anomaly detection model includes an encoding network and a decoding network; the detection module is used for:

[0035] The superpixel fusion features of the superpixel region are encoded through the coding network to obtain superpixel coded features;

[0036] The superpixel coding features of the superpixel region are decoded using the decoding network to obtain the superpixel decoding features;

[0037] By fusing the superpixel encoding features and the superpixel decoding features, a fourth superpixel feature is obtained;

[0038] Based on the fourth superpixel feature, the anomaly detection result of the superpixel region is determined.

[0039] Optionally, the encoding network includes m encoding layers connected in sequence, and the decoding network includes m decoding layers connected in sequence, where m is an integer greater than 1; the detection module is used for:

[0040] The superpixel fusion features are input into the coding network to obtain the superpixel coding features output by m coding layers in the coding network;

[0041] The superpixel coding features of the superpixel region are decoded through the decoding network to obtain the superpixel decoding features:

[0042] The superpixel encoded features output from the m-th encoding layer are input into the first decoding layer to obtain the superpixel decoded features output from the first decoding layer.

[0043] The superpixel encoded features output from the nth encoding layer and the superpixel decoded features output from the mnth decoding layer are input into the (m+1)nth decoding layer to obtain the superpixel decoded features output from the (m+1)nth decoding layer, where n is a positive integer less than m.

[0044] Optionally, the detection module is used for:

[0045] The fourth superpixel feature is obtained by fusing the superpixel coding features output from the first coding layer, the superpixel coding features output from the m-th coding layer, the superpixel decoding features output from the first decoding layer, and the superpixel decoding features output from the m-th decoding layer.

[0046] Optionally, the anomaly detection model includes an attention network and a classification network; the detection module is used for:

[0047] By multiplying the fourth superpixel feature with the mapping matrix through the attention network, we obtain the superpixel query feature, the superpixel key feature, and the superpixel value feature.

[0048] The fifth superpixel feature is obtained by fusing the superpixel query feature, the superpixel key feature, and the superpixel value feature through the attention network.

[0049] The classification network is used to transform the features of the fifth superpixel to obtain the anomaly detection result of the superpixel region. The classification network is used to predict the probability of an anomaly.

[0050] Optionally, the determining module is configured to:

[0051] Extract k first image features at different scales from the image;

[0052] By fusing k different scales of the first image features of the image, the image fusion features of the image are obtained;

[0053] Based on the image fusion features of the image, the segmentation result of the image is determined, wherein the segmentation result refers to the result of superpixel segmentation of the image.

[0054] Optionally, the superpixel segmentation model includes k feature extraction networks of different scales, each of which includes multiple feature extraction layers connected in sequence; the determining module is used to:

[0055] The image is input into k feature extraction networks in the superpixel segmentation model to obtain the image features output by the first feature extraction layer and the image features output by the last feature extraction layer in the k feature extraction networks.

[0056] For each of the feature extraction networks, the image features output by the first feature extraction layer and the image features output by the last feature extraction layer are fused to obtain k first image features at different scales.

[0057] Optionally, the image anomaly detection device is implemented through an anomaly detection model, and the device further includes a model training module for:

[0058] Determine multiple superpixel regions of a sample image, wherein the superpixel regions include multiple pixels in the sample image;

[0059] Using the anomaly detection model, k first sample superpixel features of different scales are extracted from each sample superpixel region;

[0060] By fusing the k different scales of the first sample superpixel features of each sample superpixel region, the sample superpixel fusion feature of each sample superpixel region is obtained;

[0061] Using the anomaly detection model, based on the sample superpixel fusion features of each sample superpixel region, the sample anomaly detection result for each sample superpixel region is determined respectively;

[0062] The anomaly detection model is trained based on the anomaly detection label of each superpixel region of the sample and the anomaly detection result of the sample. The anomaly detection label indicates whether an anomaly truly exists in the superpixel region of the sample.

[0063] Optionally, the model training module is used for:

[0064] Using a superpixel segmentation model, k first sample image features at different scales are extracted from the sample image.

[0065] The sample image fusion feature of the sample image is obtained by fusing the first sample image features at k different scales of the sample image;

[0066] Based on the sample image fusion features of the sample image, the sample segmentation result of the sample image is determined by the superpixel segmentation model. The sample segmentation result refers to the result of superpixel segmentation of the sample image.

[0067] Optionally, the k different scales of the first sample superpixel features are obtained through the k different scales of the feature extraction network in the anomaly detection model, and the k different first sample image features are obtained through the k different scales of the feature extraction network in the superpixel segmentation model. The k different scales of the feature extraction network in the anomaly detection model are the same as the k different scales of the feature extraction network in the superpixel segmentation model.

[0068] The model training module is used for:

[0069] Based on the anomaly detection label of each superpixel region of the sample, the anomaly detection result of the sample, the segmentation label of the sample image, and the sample segmentation result, the anomaly detection model and the superpixel segmentation model are trained, where the segmentation label refers to the result of superpixel segmentation of the sample image.

[0070] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the image anomaly detection method as described above.

[0071] On the other hand, a computer-readable storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to perform the operations performed by the image anomaly detection method as described above.

[0072] On the other hand, a computer program product is provided, including a computer program loaded and executed by a processor to perform the operations performed by the image anomaly detection method as described above.

[0073] The solution provided in this application identifies multiple superpixel regions of an image to be detected, and performs local feature extraction and fusion on a per-superpixel-region basis to identify the probability of anomalies in each superpixel region. This reduces unnecessary global computation, lowers computational complexity, eliminates the need for manual review, and improves the efficiency of anomaly detection. Furthermore, by extracting and fusing features from multiple superpixel regions at different scales, it is beneficial to capture features at different scales, ensuring the robustness and accuracy of anomaly detection. Moreover, detecting on a per-superpixel-region basis can effectively identify local anomalies in the image, improving the precision of anomaly detection. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application;

[0076] Figure 2 This is a flowchart of an image anomaly detection method provided in an embodiment of this application;

[0077] Figure 3 This is a schematic diagram of the structure of an anomaly detection model provided in an embodiment of this application;

[0078] Figure 4 This is a flowchart of another image anomaly detection method provided in the embodiments of this application;

[0079] Figure 5 This is a schematic diagram of an abnormal image provided in an embodiment of this application;

[0080] Figure 6 This is a flowchart of another image anomaly detection method provided in the embodiments of this application;

[0081] Figure 7 This is a flowchart of a model training method provided in an embodiment of this application;

[0082] Figure 8 This is a flowchart of another image anomaly detection method provided in the embodiments of this application;

[0083] Figure 9 This is a schematic diagram of the structure of an image anomaly detection device provided in an embodiment of this application;

[0084] Figure 10 This is a schematic diagram of another image anomaly detection device provided in an embodiment of this application;

[0085] Figure 11 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0086] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0088] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first superpixel feature may be referred to as a second superpixel feature, and similarly, a second superpixel feature may be referred to as a first superpixel feature.

[0089] "At least one" refers to one or more superpixel regions. For example, at least one superpixel region can be one superpixel region, two superpixel regions, three superpixel regions, or any integer number of superpixel regions greater than or equal to one. "Multiple" refers to two or more superpixel regions. For example, multiple superpixel regions can be two superpixel regions, three superpixel regions, or any integer number of superpixel regions greater than or equal to two. "Each" refers to each of the at least one superpixel regions. For example, each superpixel region refers to each of the multiple superpixel regions. If the multiple superpixel regions are three superpixel regions, then each superpixel region refers to each of the three superpixel regions.

[0090] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have all been fully authorized by the user or relevant parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0091] For example, the images, sample images, anomaly detection labels, and segmentation labels involved in this application are all fully authorized by the user or relevant parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0092] The image anomaly detection method provided in this application can be used in computer devices. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. This application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0093] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application. See also... Figure 1 The computer system includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0094] Terminal 101 has a client 111 installed and running. This client 111 can be a social application client, online payment client, online shopping client, game client, medical service client, video client, etc. When terminal 101 runs client 111, the user interface of client 111 is displayed on the screen of terminal 101. Terminal 101 is the terminal used by user 121.

[0095] Optionally, terminal 101 may refer to one of a number of terminals, including: smartphones, tablets, laptops, desktop computers, smart voice interaction devices, smart home appliances, in-vehicle terminals, aircraft, VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc., but not limited to these.

[0096] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be six, eight, or more terminals. This application does not limit the number of terminals or the type of device.

[0097] Figure 1Only one terminal is shown in the diagram, but in different embodiments, multiple other terminals 103 can access the server 102. Optionally, one or more terminals 103 may also be terminals corresponding to developers, on which a client development and editing platform is installed. Developers can edit and update the client on the terminal 103 and transmit the updated client installation package to the server 102 via wired or wireless network. Terminal 101 can download the client installation package from the server 102 to update the client.

[0098] Terminal 101 and other terminals 103 are connected to server 102 via wired or wireless networks.

[0099] Server 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 102 is used to provide backend services to clients. Optionally, server 102 undertakes the primary computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the primary computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0100] In this embodiment, terminal 101 sends an anomaly detection request for an image to server 102. Upon receiving the anomaly detection request, server 102 acquires the image to be detected. Server 102 segments the image into multiple superpixel regions, determines the anomaly detection results for each superpixel region, and the anomaly detection results represent the probability that an anomaly exists in the superpixel region. Server 102 returns the anomaly detection results for the multiple superpixel regions of the image to terminal 101.

[0101] It should be noted that the above implementation environment is only an example. The method provided in this application embodiment can also be executed by terminal 101 or server 102 alone, or by other computer devices. This application embodiment does not limit this.

[0102] Figure 2 This is a flowchart of an image anomaly detection method provided in an embodiment of this application. This embodiment is executed by a computer device. See also... Figure 2 The method includes:

[0103] 201. A computer device determines multiple superpixel regions of an image, whereby a superpixel region includes multiple pixels in the image.

[0104] The computer device acquires the image to be detected. The purpose of this application embodiment is to detect the probability that the image has anomalies. An image anomaly refers to problems such as color distortion, blurry content, disordered content, flickering, stripes, or garbled text in the image.

[0105] The image can be any type of image obtained through any means, such as an image captured from a game screen or an image captured from a live video stream.

[0106] The computer device identifies multiple superpixel regions of the image. Optionally, a superpixel region is composed of multiple pixels in the image that are adjacent in position and have similarity, and the multiple pixels in the same superpixel region have similar features in terms of color, brightness, or texture.

[0107] 202. The computer device extracts k first superpixel features of different scales from each superpixel region, where k is an integer greater than 1.

[0108] Taking one superpixel region among multiple superpixel regions as an example, the computer device extracts k first superpixel features of different scales from this superpixel region. That is, the computer device obtains k first superpixel features of this superpixel region, and the scales of the k first superpixel features are all different. Then, for each superpixel region among the multiple superpixel regions, the computer device performs the above steps to obtain k first superpixel features of different scales for each superpixel region.

[0109] In this system, the first superpixel features at different scales are obtained through convolution with kernels of different sizes. The size of the convolution kernel refers to its width and height. The width and height of the kernel together define the range of the convolution operation and the size of the receptive field. The receptive field is the area that the kernel can perceive on the input data during the convolution operation. A larger kernel results in a larger receptive field, capturing more global features. A smaller kernel results in a smaller receptive field, capturing more detailed features. Therefore, the scale of the first superpixel feature describes its performance within different receptive fields, i.e., the size of the area perceived during the convolution operation. For small-scale first superpixel features, the feature can capture detailed information about the superpixel region, such as subtle textures, local color variations, local edges, or the outlines of small objects. For large-scale first superpixel features, the feature can capture global information about the superpixel region, such as global texture, overall color distribution, and the shape or outline of large objects.

[0110] 203. The computer device fuses the first superpixel features of k different scales in each superpixel region to obtain the superpixel fused features of each superpixel region.

[0111] Taking one superpixel region among multiple superpixel regions as an example, after acquiring k first superpixel features of different scales for that superpixel region, the computer device fuses these k first superpixel features to obtain the superpixel fused feature of that superpixel region. Then, for each superpixel region among the multiple superpixel regions, the computer device performs the above steps to obtain the superpixel fused feature of each superpixel region.

[0112] Among them, the first superpixel features at different scales can provide complementary information. By fusing the first superpixel features at different scales, the resulting superpixel fusion feature contains both detailed and global information. Therefore, the superpixel fusion feature can more accurately and comprehensively reflect the characteristics of the superpixel region.

[0113] 204. The computer device determines the anomaly detection result for each superpixel region based on the superpixel fusion features of each superpixel region. The anomaly detection result represents the probability that there is an anomaly in the superpixel region of the image.

[0114] Taking one superpixel region among multiple superpixel regions as an example, the computer device analyzes the probability of an anomaly in that superpixel region based on its superpixel fusion features, thus obtaining the anomaly detection result for that superpixel region. Then, for each superpixel region, the computer device performs the above steps to obtain the anomaly detection result for each superpixel region. Furthermore, based on the anomaly detection result for each superpixel region, it determines which regions in the image contain anomalies and which do not, thereby achieving anomaly detection in the image.

[0115] The method provided in this application identifies multiple superpixel regions of an image to be detected, and performs local feature extraction and fusion on a superpixel region basis to identify the probability of anomalies in each superpixel region. This reduces unnecessary global computation, lowers computational complexity, eliminates the need for manual review, and improves the efficiency of anomaly detection. Furthermore, by extracting and fusing features from multiple superpixel regions at different scales, it is beneficial to capture features at different scales, ensuring the robustness and accuracy of anomaly detection. Moreover, detecting on a superpixel region basis can effectively identify local anomalies in the image, improving the precision of anomaly detection.

[0116] The above Figure 2 The embodiments described herein are merely brief illustrations of image anomaly detection methods. In some embodiments, the computer device may also perform the image anomaly detection process using an anomaly detection model.

[0117] Figure 3 This is a schematic diagram of the structure of an anomaly detection model shown in an embodiment of this application, such as... Figure 3 As shown, the anomaly detection model includes k feature extraction networks, encoding networks, decoding networks, attention networks, and classification networks at different scales. The k feature extraction networks at different scales are connected to the encoding network, the encoding network is connected to the decoding network, the decoding network is connected to the attention network, and the attention network is connected to the classification network.

[0118] Among them, k feature extraction networks of different scales are used to extract superpixel features at different scales. For example... Figure 3 As shown, each feature extraction network consists of multiple feature extraction layers connected in sequence. The encoding network is used to encode the features, such as... Figure 3 As shown, the encoding network consists of m sequentially connected encoding layers. The decoding network is used to decode the features, such as... Figure 3 As shown, the decoding network consists of m decoding layers connected in sequence. The attention network is used to process the features using an attention mechanism, and the classification network is used to predict the probability of whether the input features are abnormal.

[0119] The specific process of computer equipment performing anomaly detection on images using an anomaly detection model is described below. Figure 4 Examples of implementations. Figure 4 This is a flowchart of another image anomaly detection method provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 4 The method includes:

[0120] 401. A computer device determines multiple superpixel regions of an image, whereby a superpixel region includes multiple pixels in the image.

[0121] Figure 3 The example shown is an image containing seven superpixel regions. The image includes superpixel regions 1 to 7, and the computer device can determine the seven superpixel regions of the image.

[0122] In one possible implementation, a computer device acquires an image to be detected, preprocesses the image to be detected, obtains a preprocessed image, and determines multiple superpixel regions of the preprocessed image.

[0123] Optionally, preprocessing includes format conversion. For example, the image is an RGB image, where the pixel values ​​of each pixel consist of the color values ​​of the red, green, and blue channels. For ease of calculation, the computer converts the RGB image to a grayscale image. Compared to an RGB image, although a grayscale image has less data, it retains the main edge and shape information of the RGB image. The conversion formula for converting an RGB image to a grayscale image is shown in formula (1) below.

[0124] Igray =0.299·I R +0.587·I G +0.114·I B ; Formula (1)

[0125] Among them, I R I represents the color value of a pixel in the red channel of an RGB image. G I represents the color value of a pixel in the green channel of an RGB image. B I represents the color value of a pixel in the blue channel of an RGB image. gray This represents the pixel value of a pixel in a grayscale image.

[0126] Optionally, preprocessing includes normalization. The purpose of normalization is to scale the pixel values ​​of the image to a specific range, such as [0,1] or [-1,1].

[0127] For example, the formula for normalizing an image by a computer device is as follows (2).

[0128]

[0129] Where I represents the pixel value before normalization, I ′ I represents the normalized pixel value. min I represents the minimum pixel value in the image before normalization. max This represents the maximum pixel value in the image before normalization.

[0130] For example, the formula for normalizing an image by a computer device is as follows (3).

[0131]

[0132] Where I represents the pixel value before normalization, I ′ σ represents the normalized pixel value, μ represents the mean of multiple pixel values ​​in the image before normalization, and σ represents the variance of multiple pixel values ​​in the image before normalization.

[0133] In one possible implementation, multiple superpixel regions of the image are determined, including either of the following two methods.

[0134] The first method involves using a computer device to determine multiple cluster centers in an image; clustering multiple pixels in the image based on these centers to obtain multiple clusters; and defining the region formed by multiple pixels in any cluster as a superpixel region.

[0135] In this context, cluster centers are pixels in an image. For any other pixel in the image besides the cluster centers, the computer device assigns the pixel to the cluster containing a certain cluster center, thus obtaining multiple clusters. Each cluster includes a cluster center and at least one pixel assigned to that cluster.

[0136] Optionally, the computer device determines multiple initial cluster centers in the image, for example, by random selection or other methods. The computer device assigns multiple pixels in the image to the cluster containing a given cluster center. For each cluster, the computer device determines the mean of all pixels (including the cluster center) within that cluster; this mean could be the mean of pixel values ​​or the mean of pixel features. The pixel in the cluster closest to the mean is selected as the new cluster center. The steps of assigning pixels and updating cluster centers are iteratively executed until an iteration termination condition is met, thus obtaining the final cluster centers. This iteration termination condition could be that the cluster centers no longer change, or the number of iterations reaches a preset number, etc.

[0137] In this implementation, the pixels in the image are clustered around the cluster center point. This can group pixels with similar features into the same superpixel region, which helps to preserve the local consistency of the image, making the segmentation results more accurate, conforming to the true structure of the image, and reducing edge blurring and inconsistencies within the region.

[0138] Optionally, the computer device clusters multiple pixels in the image based on multiple cluster centroids to obtain multiple clusters, including: for any pixel, determining the feature distance between the pixel and the multiple cluster centroids based on the color distance and spatial distance between the pixel and the multiple cluster centroids, where the color distance refers to the distance between the color parameters of the pixel and the color parameters of the cluster centroids, and the spatial distance refers to the distance between the position parameters of the pixel and the position parameters of the cluster centroids; and assigning the pixel to the cluster containing the cluster centroid with the smallest feature distance to the pixel.

[0139] For example, a computer device uses the following formula (4) to determine the color distance between pixels.

[0140]

[0141] Here, i and j represent two different pixels. L i a i and b i L represents the color parameter of pixel i in the Lab color space. j a j and b jThis represents the color parameter of pixel j in the Lab color space. i a represents the brightness of pixel i in the Lab color space. i b represents the hue of pixel i in the Lab color space. i This represents the saturation of pixel i in the Lab color space. j a represents the brightness of pixel j in the Lab color space. j b represents the hue of pixel j in the Lab color space. j dc represents the saturation of pixel j in the Lab color space. dc represents the color distance between pixel i and pixel j.

[0142] For example, a computer device uses the following formula (5) to determine the spatial distance between pixels.

[0143]

[0144] Here, i and j represent two different pixels. i and y i The position parameter representing pixel i, x i Represents the x-coordinate of pixel i on the x-axis, y-coordinate of pixel i on the y-axis. i This represents the y-coordinate of pixel i. j and y j The position parameter representing pixel j, x j This represents the x-coordinate of pixel j on the x-axis, and y-coordinate of pixel j on the y-axis. j This represents the y-coordinate of pixel j. s This represents the spatial distance between pixel i and pixel j.

[0145] For example, a computer device uses the following formula (6) to determine the feature distance between pixels.

[0146]

[0147] Where, d c d represents the color distance between pixel i and pixel j. s N represents the spatial distance between pixel i and pixel j. c and N s These are the normalization factors for color distance and spatial distance, respectively. D represents the feature distance between pixel i and pixel j.

[0148] In this implementation, the feature distance between pixels is obtained by combining the color distance and spatial distance between pixels. Since the feature distance considers both the color similarity and spatial proximity of pixels in the image, it avoids the segmentation inconsistencies caused by relying solely on color features, thus segmenting more coherent superpixel regions. Furthermore, by considering both color and spatial information, this method can more accurately group similar and adjacent pixels together while separating pixels with significant differences, reducing over-segmentation and blurred boundaries, thereby improving segmentation accuracy.

[0149] The second method is to extract k different scales of first image features from the image; fuse the k different scales of first image features to obtain the image fusion features; and determine the image segmentation result based on the image fusion features, where the segmentation result refers to the result of superpixel segmentation of the image.

[0150] The scale of the first image feature describes its performance within different receptive fields, i.e., the size of the region perceived during the convolution operation. For small-scale first image features, they can capture detailed information about the image, such as subtle textures, local color variations, local edges, or the outlines of small objects. For large-scale first image features, they can capture global information about the image, such as global texture, overall color distribution, and the shape or outline of large objects.

[0151] The computer device analyzes the image based on its image fusion features to determine how to segment the image, obtaining a segmentation result that indicates how to divide the image into multiple superpixel regions. For example, the segmentation result includes the probabilities of multiple pixels in the image, where each pixel's probability represents the probability that it belongs to the edge of a superpixel region. Based on this segmentation result, the edges of superpixel regions can be determined in the image, thereby segmenting the image into multiple superpixel regions.

[0152] In this implementation, by extracting image features at different scales, it is possible to capture detailed information at different levels in the image, such as local details, textures, and global structure. By fusing these multi-scale features, the local and global information of the image can be integrated, improving the accuracy and precision of superpixel segmentation and enabling better identification and segmentation of superpixel regions.

[0153] Optionally, the superpixel segmentation model includes k feature extraction networks at different scales, each feature extraction network including multiple feature extraction layers connected in sequence; extracting the first image features at k different scales of the image includes: inputting the image into the k feature extraction networks in the superpixel segmentation model respectively, to obtain the image features output by the first feature extraction layer and the image features output by the last feature extraction layer in the k feature extraction networks; for each feature extraction network, fusing the image features output by the first feature extraction layer and the image features output by the last feature extraction layer in the feature extraction network to obtain the first image features at k different scales respectively.

[0154] In this feature extraction network, the feature extraction layer can be a convolutional layer. The different scales of the feature extraction network refer to the different scales of the convolutional layers, which in turn refer to the different sizes of the convolutional kernels.

[0155] Taking one of the k feature extraction networks as an example, this feature extraction network includes multiple feature extraction layers connected in sequence. The computer device inputs an image into the first feature extraction layer of this network to obtain the image features output by the first feature extraction layer. The image features output by the first feature extraction layer are then input into the second feature extraction layer to obtain the image features output by the second feature extraction layer. The image features output by the second feature extraction layer are then input into the third feature extraction layer to obtain the image features output by the third feature extraction layer, and so on, until the image features output by the last feature extraction layer are obtained. In this embodiment, the computer device fuses the image features output by the first feature extraction layer and the image features output by the last feature extraction layer to obtain a first image feature.

[0156] The computer device obtains a first image feature through each feature extraction network according to the above method, thereby obtaining k first image features. Since the scales of the k feature extraction networks are different, the scales of the k first image features are also different.

[0157] In this implementation, by fusing the image features output from the first and last feature extraction layers of the feature extraction network, both low-level and high-level feature information can be comprehensively considered. The image features output from the first feature extraction layer contain detailed information, while the image features output from the last feature extraction layer contain high-level semantic information. This multi-level information fusion approach can provide a more comprehensive feature representation for superpixel segmentation, which is beneficial for improving the accuracy of superpixel segmentation.

[0158] Optionally, the computer device determines the image segmentation result based on the image fusion features of the image, including: encoding the image fusion features of the image to obtain image coded features; decoding the image coded features to obtain image decoded features; fusing the superpixel coded features and the superpixel decoded features to obtain a second image feature; and determining the image segmentation result based on the second image feature.

[0159] Optionally, the superpixel segmentation model includes k feature extraction networks, encoding networks, decoding networks, and classification networks at different scales. The computer device encodes the image fusion features of the image through the encoding network to obtain image encoded features, and decodes the image encoded features through the decoding network to obtain image decoded features. The computer device fuses the superpixel encoded features and superpixel decoded features to obtain second image features. The computer device then transforms the second image features through the classification network to obtain the image segmentation result.

[0160] For example, the feature extraction networks at k different scales in the superpixel segmentation model are the same as those at k different scales in the anomaly detection model. However, the other network structures in the superpixel segmentation model are different from those in the anomaly detection model. In other words, the encoding, decoding, and classification networks in the superpixel segmentation model are different from those in the anomaly detection model.

[0161] In one possible implementation, after the computer device segments the image into multiple superpixel regions, it can also merge and optimize the multiple superpixel regions obtained from the initial segmentation to ensure that the superpixel regions have good connectivity and consistency.

[0162] For example, a computer device uses an image-based clustering method to merge similar and adjacent superpixel regions. In this image-based clustering method, each superpixel region is treated as a node in the image, and the edge weights between nodes are determined by the similarity between the superpixel regions. Two adjacent superpixel regions with edge weights greater than a preset edge weight are merged.

[0163] For example, a computer device uses an edge-based information merging method to merge similar and adjacent superpixel regions. In this method, the computer device determines the edge strength between adjacent superpixel regions and merges two adjacent superpixel regions whose edge strength is less than a preset edge strength.

[0164] 402. The computer device extracts k first superpixel features of different scales from each superpixel region through k feature extraction networks of different scales in the anomaly detection model, where k is an integer greater than 1.

[0165] In this embodiment, a computer device implements an image anomaly detection method using an anomaly detection model. This anomaly detection model is used to detect the probability of anomalies existing in an image or superpixel region. The anomaly detection model includes k feature extraction networks of different scales. The scale of the feature extraction network refers to the size of the convolutional kernels in the feature extraction network; the size of the convolutional kernels differs in feature extraction networks of different scales.

[0166] In one possible implementation, each feature extraction network includes multiple feature extraction layers connected in sequence. Step 402 includes: inputting the superpixel region into k feature extraction networks in the anomaly detection model respectively, to obtain the superpixel features output by the first feature extraction layer and the superpixel features output by the last feature extraction layer in the k feature extraction networks; for each feature extraction network, fusing the superpixel features output by the first feature extraction layer and the superpixel features output by the last feature extraction layer in the feature extraction network to obtain k first superpixel features at different scales.

[0167] Taking one of the k feature extraction networks as an example, this feature extraction network includes multiple feature extraction layers connected in sequence. The computer device inputs a superpixel region into the first feature extraction layer of the network to obtain the superpixel feature output by the first feature extraction layer. The superpixel feature output by the first feature extraction layer is then input into the second feature extraction layer to obtain the superpixel feature output by the second feature extraction layer. The superpixel feature output by the second feature extraction layer is then input into the third feature extraction layer to obtain the superpixel feature output by the third feature extraction layer, and so on, until the superpixel feature output by the last feature extraction layer is obtained. In this embodiment, the computer device fuses the superpixel feature output by the first feature extraction layer and the superpixel feature output by the last feature extraction layer to obtain a first superpixel feature.

[0168] The computer device obtains a first superpixel feature through each feature extraction network according to the above method, thereby obtaining k first superpixel features. Since the scales of the k feature extraction networks are different, the scales of the k first superpixel features are also different.

[0169] Optionally, the processing of the feature extraction layer can be represented by the following formula (7).

[0170]

[0171] Among them, S i+m，j+n，c W represents the superpixel region. m，n，c，k b represents the convolutional kernel of the feature extraction layer k The parameter represents the bias parameter, M and N represent the height and width of the convolution kernel, respectively, C represents the number of input channels, and k represents the output channel index. Fi，j，k This represents the superpixel features output by the feature extraction layer.

[0172] Optionally, the computer device uses the following formula (8) to fuse the superpixel features output by the first feature extraction layer and the superpixel features output by the last feature extraction layer.

[0173] F fusion =[F l F m ]; Formula (8)

[0174] Among them, F l F represents the superpixel features output by the first feature extraction layer. m F represents the superpixel feature output by the last feature extraction layer, and [·] represents the feature concatenation operation. fusion This represents the first superpixel feature.

[0175] In superpixel feature extraction, texture features are crucial, representing information such as repetitive patterns and surface structures within the superpixel region. Feature extraction layers effectively capture these texture features. To extract rich texture features, computer devices use multiple sequentially connected feature extraction layers. Shallow layers capture shallow texture features, such as edges and corners, while deeper layers capture deep texture features, such as shapes and patterns. Therefore, the first superpixel feature, obtained by fusing the superpixel features output from the first and last feature extraction layers, combines both shallow and deep texture features.

[0176] In this implementation, by fusing the superpixel features from the first and last feature extraction layers, both low-level detail information and high-level semantic information of the superpixel region can be obtained simultaneously. This improves the accuracy of the extracted superpixel features and helps to better identify the probability of anomalies in the superpixel region. Furthermore, this combination of multi-level features helps maintain high detection accuracy and robustness in complex scenes, thereby improving the generalization ability in different scenarios.

[0177] 403. For each superpixel region, the computer device fuses k first superpixel features of different scales in the superpixel region to obtain the second superpixel feature of the superpixel region, determines the adjacent superpixel regions of the superpixel region, and fuses the second superpixel feature of the superpixel region and the second superpixel feature of the adjacent superpixel regions to obtain the superpixel fused feature of the superpixel region.

[0178] In this context, the adjacent superpixel regions of a superpixel region refer to other superpixel regions that are adjacent to the superpixel region, that is, other superpixel regions that are directly connected to the superpixel region. A superpixel region can have one or more adjacent superpixel regions.

[0179] Taking one superpixel region among multiple superpixel regions as an example, the computer device determines the neighboring superpixel regions of this superpixel region, and fuses the second superpixel features of this superpixel region with the second superpixel features of the neighboring superpixel regions to obtain the superpixel fused feature of this superpixel region. Here, the second superpixel feature of the neighboring superpixel regions refers to the feature obtained by fusing k first superpixel features of neighboring superpixel regions at different scales.

[0180] In this implementation, by fusing the superpixel features of a superpixel region with those of adjacent superpixel regions, spatial consistency can be enhanced by leveraging neighborhood information. This makes the superpixel features of the superpixel region more locally coherent, improving spatial consistency and thus enhancing the robustness of anomaly detection. Furthermore, by fusing the superpixel features of adjacent superpixel regions, the noise influence that may occur in a single superpixel region can be smoothed out. This ensures that the superpixel features of each superpixel region not only depend on themselves but also reference the features of adjacent superpixel regions, thereby reducing errors caused by noise and improving overall detection accuracy.

[0181] In one possible implementation, the computer device fuses k different scales of first superpixel features of the superpixel region to obtain second superpixel features of the superpixel region, including: weighted summation of the k different scales of the first superpixel features of the superpixel region to obtain second superpixel features of the superpixel region.

[0182] Optionally, each scale of the first superpixel feature has its own weight, which can be a pre-set value. The computer device uses k weights to perform a weighted summation of the k different scales of the first superpixel features of the superpixel region to obtain the second superpixel feature of the superpixel region.

[0183] For example, with k equal to 2, the computer device uses the following formula (9) to perform a weighted summation of the first superpixel features at two different scales.

[0184] F multi =αF s1 +βF s2 ; Formula (9)

[0185] Where s1 and s2 represent two different scales, F s1 F represents the first superpixel feature at scale s1. s2Let F represent the first superpixel feature at scale s2, where α and β are the weights at scale s1 and scale s2, respectively. multi This represents the second superpixel feature.

[0186] In one possible implementation, the computer device determines the weight between the superpixel region and the adjacent superpixel region based on the difference between the second superpixel feature of the superpixel region and the second superpixel feature of the adjacent superpixel region, wherein the weight is negatively correlated with the difference; based on the weight between the superpixel region and the adjacent superpixel region, the second superpixel feature of the adjacent superpixel region is weighted to obtain the third superpixel feature of the adjacent superpixel region; the second superpixel feature of the superpixel region is added to the third superpixel feature of the adjacent superpixel region to obtain the superpixel fusion feature of the superpixel region.

[0187] Optionally, if there are multiple adjacent superpixel regions, the computer device determines the weight of each adjacent superpixel region, and weights the multiple adjacent superpixel regions based on their respective weights to obtain the third superpixel feature of each adjacent superpixel region. The second superpixel feature of the superpixel region is then added to the third superpixel feature of the multiple adjacent superpixel regions to obtain the superpixel fusion feature of the superpixel region.

[0188] For example, the computer device uses the following formula (10) to determine the weights between superpixel regions and adjacent superpixel regions.

[0189]

[0190] Among them, w ij F represents the weight between the i-th superpixel region and its j-th adjacent superpixel region. i F represents the second superpixel feature of the i-th superpixel region. j Let represent the second superpixel feature of the j-th adjacent superpixel region of the i-th superpixel region. σ represents a parameter controlling the weight decay rate.

[0191] For example, the computer device uses the following formula (11) to determine the superpixel fusion features of the superpixel region.

[0192]

[0193] in, F represents the superpixel fusion feature of the i-th superpixel region. i F represents the second superpixel feature of the i-th superpixel region. j Let w represent the second superpixel feature of the j-th adjacent superpixel region among i superpixel regions. ijThe weight between the i-th superpixel region and its j-th adjacent superpixel region is represented by N(i), and N(i) represents at least one adjacent superpixel region of the i-th superpixel region.

[0194] In this implementation, the weights between a superpixel region and its neighboring superpixel regions are negatively correlated with the feature difference between them. Therefore, a higher feature similarity to the superpixel region results in a higher weight. This weighting method allows for more thorough feature fusion between the superpixel region and its more similar neighboring superpixel regions, thereby improving the accuracy of feature fusion based on neighboring superpixel regions. Furthermore, when the similarity between a neighboring superpixel region and the current superpixel region is low, the neighboring superpixel region has a lower weight, reducing the degree of feature fusion with that neighboring superpixel region. This avoids the influence of dissimilar neighboring superpixel regions on the current superpixel region, further improving feature accuracy.

[0195] 404. The computer device encodes the superpixel fusion features of the superpixel region through the coding network in the anomaly detection model to obtain the superpixel coded features.

[0196] The anomaly detection model includes an encoding network used for feature encoding. The computer device inputs the superpixel fused features of the superpixel region into the encoding network to obtain the superpixel encoded features.

[0197] In one possible implementation, the encoding process can be represented by the following formula (12).

[0198] E = Unetf enc (S); Formula (12)

[0199] Where E represents superpixel coding features, S represents superpixel fusion features, and Unef represents superpixel fusion features. enc (·) indicates a coding network.

[0200] In one possible implementation, the encoding network comprises m encoding layers connected sequentially, where m is an integer greater than 1. Then step 404 is replaced by step 4041.

[0201] 4041. Input the superpixel fusion features into the coding network to obtain the superpixel coding features output by m coding layers in the coding network.

[0202] The coding network comprises m coding layers connected in sequence. The computer device inputs the superpixel fusion feature into the first coding layer of the coding network to obtain the superpixel coding feature output by the first coding layer. The superpixel coding feature output by the first coding layer is then input into the second coding layer to obtain the superpixel coding feature output by the second coding layer. The superpixel coding feature output by the second coding layer is then input into the third coding layer to obtain the superpixel coding feature output by the third coding layer, and so on, until the superpixel feature output by the last coding layer (i.e., the m-th coding layer) is obtained.

[0203] Alternatively, the coding layer can be a convolutional layer.

[0204] 405. The computer device decodes the superpixel coding features of the superpixel region through the decoding network in the anomaly detection model to obtain the superpixel decoding features.

[0205] The anomaly detection model includes a decoding network, which is used for feature decoding. The computer device inputs the superpixel encoded features into the decoding network to obtain the superpixel decoded features.

[0206] In one possible implementation, the decoding process can be represented by the following formula (13).

[0207] D = Unetf dec (E); Formula (13)

[0208] Where D represents the superpixel decoding feature, E represents the superpixel encoding feature, and Unetf dec (·) indicates the decoding network.

[0209] In this embodiment, an encoding and decoding structure is introduced into the anomaly detection model to encode and decode the features of the superpixel region, which can capture more texture information. Furthermore, the features obtained through the encoding and decoding operations are more compact, which is beneficial for more accurate anomaly detection.

[0210] In one possible implementation, the encoding network comprises m sequentially connected encoding layers, and the decoding network comprises m sequentially connected decoding layers, where m is an integer greater than 1. Step 404 is replaced by step 4041, and therefore step 405 is replaced by step 4051 as described below.

[0211] 4051. Input the superpixel encoded features output by the m-th encoding layer into the 1st decoding layer to obtain the superpixel decoded features output by the 1st decoding layer; input the superpixel encoded features output by the n-th encoding layer and the superpixel decoded features output by the mn-th decoding layer into the (m+1)-n-th decoding layer to obtain the superpixel decoded features output by the (m+1)-n-th decoding layer, where n is a positive integer less than m.

[0212] The input to the first decoding layer is the superpixel encoded feature output by the m-th encoding layer. The input to the second decoding layer is the superpixel encoded feature output by the (m-1)-th encoding layer and the superpixel decoded feature output by the first decoding layer. The input to the third decoding layer is the superpixel encoded feature output by the (m-2)-th encoding layer and the superpixel decoded feature output by the second decoding layer. And so on, the input to the m-th decoding layer (which is also the last decoding layer) is the superpixel encoded feature output by the first encoding layer and the superpixel decoded feature output by the (m-1)-th decoding layer.

[0213] In this implementation, the layer-by-layer matching of each encoding and decoding layer enables the layer-by-layer transmission and fusion of encoded and decoded features, which can preserve features at various scales and levels, thereby retaining richer detailed features and improving the accuracy of anomaly detection.

[0214] 406. The computer device fuses the superpixel encoding features and superpixel decoding features to obtain the fourth superpixel feature.

[0215] After obtaining the superpixel encoded features output by the encoding network and the superpixel decoded features output by the decoding network, the computer device fuses the superpixel encoded features and the superpixel decoded features to obtain the fourth superpixel feature.

[0216] In this implementation, encoding the superpixel fusion features via an encoding network extracts features containing high-level semantic information, thus better representing the potential features of the superpixel region. The decoding network recovers and enhances spatial detail information, enabling the anomaly detection model to learn more anomaly-related features during the encoding and decoding process. Furthermore, the fourth superpixel feature obtained by fusing the encoded and decoded features simultaneously contains both high-level semantic information and spatial detail information; this multi-level feature fusion approach improves the accuracy of anomaly detection.

[0217] In one possible implementation, the encoding network comprises m sequentially connected encoding layers, and the decoding network comprises m sequentially connected decoding layers, where m is an integer greater than 1. Step 404 is replaced by step 4041, and step 405 is replaced by step 4051. Therefore, step 406 is replaced by step 4061.

[0218] 4061. The superpixel coding features output by the first coding layer, the superpixel coding features output by the m-th coding layer, the superpixel decoding features output by the first decoding layer, and the superpixel decoding features output by the m-th decoding layer are fused to obtain the fourth superpixel feature.

[0219] Optionally, the computer device connects the superpixel coded features output by the first coding layer, the superpixel coded features output by the m-th coding layer, the superpixel decoded features output by the first decoding layer, and the superpixel decoded features output by the m-th decoding layer to obtain the fourth superpixel feature.

[0220] Optionally, the computer device concatenates the superpixel coded features output from the first coding layer, the superpixel coded features output from the m-th coding layer, the superpixel decoded features output from the first decoding layer, and the superpixel decoded features output from the m-th decoding layer to obtain a sixth superpixel feature. The sixth superpixel feature is then transformed to obtain a fourth superpixel feature. For example, an attention mechanism can be used to process the sixth superpixel feature.

[0221] For example, the computer device uses the following formula (14) to obtain the fourth superpixel feature.

[0222] F final =[E l E m D l D m ]; Formula (14)

[0223] Among them, F final E represents the fourth superpixel feature. l E represents the superpixel encoded feature output by the first coding layer. m D represents the superpixel coded feature output by the m-th coding layer. l D represents the superpixel decoding feature output by the first decoding layer. m This represents the superpixel decoding feature output by the m-th decoding layer. [·] indicates the feature concatenation operation.

[0224] In this implementation, the encoding and decoding features of the first layer contain detailed information, while the encoding and decoding features of the last layer contain high-level semantic information. By fusing these two layers of features, the resulting fourth superpixel feature possesses rich detailed and semantic information, enhancing the accuracy, robustness, and applicability of anomaly detection.

[0225] 407. The computer device determines the anomaly detection result of the superpixel region based on the fourth superpixel feature. The anomaly detection result represents the probability that there is an anomaly in the superpixel region of the image.

[0226] The computer equipment analyzes and processes the features of the fourth superpixel using an anomaly detection model to obtain the anomaly detection results for the superpixel region.

[0227] In one possible implementation, the probability represented by the anomaly detection result ranges from 0 to 1. If the probability of an anomaly in the superpixel region indicated by the anomaly detection result is greater than a probability threshold, then the superpixel region is determined to have an anomaly; if the probability of an anomaly in the superpixel region indicated by the anomaly detection result is not greater than the probability threshold, then the superpixel region is determined not to have an anomaly. Optionally, this probability threshold can be flexibly set according to actual conditions. For example, the probability threshold can be equal to 0.6 or 0.7, etc.

[0228] In another possible implementation, the probability represented by the anomaly detection result ranges from 0 to 1. If the probability of an anomaly in the superpixel region being anomaly is 1, then the superpixel region is determined to be anomaly; if the probability of an anomaly in the superpixel region is 0, then the superpixel region is determined not to be anomaly.

[0229] In one possible implementation, the anomaly detection model includes an attention network and a classification network. Step 407 includes steps 4071-4073.

[0230] 4071. By using an attention network, the fourth superpixel feature is multiplied by the mapping matrix to obtain the superpixel query feature, superpixel key feature, and superpixel value feature.

[0231] In this embodiment, the key feature, value feature, and query feature belong to different feature spaces. The value feature represents the feature of the fourth superpixel feature itself, while the query feature and key feature are used to determine the importance of each dimension of the features in the fourth superpixel feature.

[0232] Here, the mapping matrix is ​​the parameter matrix in the attention network, which is used to perform spatial transformation on the fourth superpixel feature.

[0233] Optionally, the mapping matrix is ​​a three-dimensional mapping matrix, including a query-dimensional mapping matrix, a key-dimensional mapping matrix, and a value-dimensional mapping matrix. The computer device multiplies each of the three-dimensional mapping matrices with the fourth superpixel feature to obtain the superpixel query feature, the superpixel key feature, and the superpixel value feature, respectively.

[0234] 4072. By using an attention network, the superpixel query features, superpixel key features, and superpixel value features are fused to obtain the fifth superpixel feature.

[0235] Optionally, the computer device normalizes the product of the superpixel query feature, the transpose of the superpixel key feature, and the scaling factor to obtain the normalized feature. The normalized feature is then multiplied by the superpixel value feature to obtain the fifth superpixel feature.

[0236] In this embodiment, an attention mechanism is introduced to process the fourth superpixel feature and finally obtain the fifth superpixel feature, which can enhance the important features in the fourth superpixel feature and suppress the unimportant features.

[0237] 4073. By transforming the features of the fifth superpixel through a classification network, the anomaly detection results of the superpixel region are obtained. The classification network is used to predict the probability of anomalies.

[0238] The computer device inputs the fifth superpixel feature into the classification network and obtains the anomaly detection result output by the classification network.

[0239] Alternatively, the classification network can be a convolutional neural network.

[0240] Optionally, the classification network includes convolutional layers, pooling layers, and fully connected layers.

[0241] For example, the processing of a convolutional layer can be represented by the following formula (15).

[0242]

[0243] Where σ represents the activation function, for example, the activation function is ReLU (Rectified Linear Unit). This represents the input of the l-th convolutional layer, which is also the output of the (l-1)-th convolutional layer. This represents the output of the l-th convolutional layer. This represents the convolution kernel of the l-th convolutional layer. This represents the bias parameters of the l-th convolutional layer. M and N represent the height and width of the convolutional kernel, respectively, C represents the number of input channels, and k represents the output channel index.

[0244] For example, the pooling layer processing can be represented by the following formula (16).

[0245]

[0246] in, This represents the input to the l-th pooling layer. This represents the output of the l-th pooling layer.

[0247] For example, the processing of a fully connected layer can be represented by the following formula (17).

[0248] Z (l) =W (l) ·F (l-1) +b (l) : Formula (17)

[0249] Among them, Z (l) F represents the output of the l-th fully connected layer.(l-1) W represents the input of the l-th fully connected layer. (l) b represents the convolutional kernel of the l-th fully connected layer. (l) This represents the bias parameter of the l-th fully connected layer.

[0250] In this implementation, the attention network generates superpixel query features, superpixel key features, and superpixel value features. It can adaptively allocate feature weights, automatically select important features, enhance the influence of key features, and suppress the interference of unimportant features, thereby improving the anomaly detection effect.

[0251] Figure 5 This is a schematic diagram of an abnormal image provided in an embodiment of this application. Figure 5 The image shown is an abnormal image, containing the garbled character 501.

[0252] Figure 6 This is a flowchart of another image anomaly detection method provided in the embodiments of this application, such as... Figure 6 As shown, the image to be processed is acquired, preprocessed, and segmented into multiple superpixel regions. For any superpixel region, a convolutional neural network is used to extract features from the superpixel region, resulting in multiple superpixel features of different scales. These multiple superpixel features of different scales are then fused to obtain superpixel fusion features. Anomaly detection is performed based on the superpixel fusion features to obtain anomaly detection results.

[0253] It should be noted that, Figure 3 The example shown is the anomaly detection result of superpixel region 1 in the image. In fact, for any one of superpixel regions 1 to 7 in the image, the result can be obtained in the following manner. Figure 3 The process shown determines the anomaly detection results.

[0254] The method provided in this application identifies multiple superpixel regions of an image to be detected, and performs local feature extraction and fusion on a superpixel region basis to identify the probability of anomalies in each superpixel region. This reduces unnecessary global computation, lowers computational complexity, eliminates the need for manual review, and improves the efficiency of anomaly detection. Furthermore, by extracting and fusing features from multiple superpixel regions at different scales, it is beneficial to capture features at different scales, ensuring the robustness and accuracy of anomaly detection. Moreover, detecting on a superpixel region basis can effectively identify local anomalies in the image, improving the precision of anomaly detection.

[0255] Figure 7 This is a flowchart illustrating a model training method provided in an embodiment of this application. This embodiment is executed by a computer device. See also... Figure 7 The method includes:

[0256] 701. A computer device determines multiple sample superpixel regions of a sample image, wherein the sample superpixel regions include multiple pixels in the sample image.

[0257] In one possible implementation, the computer device determines multiple sample cluster centers in the sample image; based on the multiple sample cluster centers, it clusters multiple sample pixels in the sample image to obtain multiple sample clusters; and the region formed by multiple sample pixels in any sample cluster is defined as a sample superpixel region.

[0258] In one possible implementation, the computer device extracts k first sample image features at different scales from the sample image using a superpixel segmentation model; fuses the k first sample image features at different scales to obtain the sample image fusion features; and determines the sample segmentation result of the sample image based on the sample image fusion features using the superpixel segmentation model. The sample segmentation result refers to the result of superpixel segmentation of the sample image.

[0259] The process of segmenting the sample image in step 701 is the same as the process of segmenting the image in step 301 above, and will not be described in detail here.

[0260] 702. The computer equipment extracts k first sample superpixel features of different scales from each sample superpixel region using an anomaly detection model.

[0261] In one possible implementation, the anomaly detection model includes k feature extraction networks at different scales, each feature extraction network comprising multiple feature extraction layers connected sequentially. Step 702 includes: inputting the sample superpixel region into the k feature extraction networks in the anomaly detection model respectively, to obtain the sample superpixel features output by the first feature extraction layer and the sample superpixel features output by the last feature extraction layer in the k feature extraction networks; for each feature extraction network, fusing the sample superpixel features output by the first feature extraction layer and the sample superpixel features output by the last feature extraction layer in the feature extraction network to obtain first sample superpixel features at k different scales.

[0262] The process of extracting the first sample superpixel feature in step 702 is the same as the process of extracting the first superpixel feature in step 302 above, and will not be described in detail here.

[0263] 703. The computer device fuses the first sample superpixel features of k different scales in each sample superpixel region to obtain the sample superpixel fusion feature of each sample superpixel region.

[0264] In one possible implementation, first sample superpixel features at k different scales of the sample superpixel region are fused to obtain second sample superpixel features of the sample superpixel region. The sample adjacent superpixel regions of the sample superpixel region are determined. The second sample superpixel features of the sample superpixel region and the second sample superpixel features of the sample adjacent superpixel regions are fused to obtain sample superpixel fusion features of the sample superpixel region.

[0265] The process of obtaining sample superpixel fusion features in step 703 is the same as the process of obtaining superpixel fusion features in step 303 above, and will not be described in detail here.

[0266] 704. The computer equipment uses an anomaly detection model to determine the anomaly detection result for each sample superpixel region based on the sample superpixel fusion features of each sample superpixel region.

[0267] In one possible implementation, the anomaly detection model includes an encoding network and a decoding network. The computer device uses the encoding network to encode the superpixel fusion features of the sample superpixel region, obtaining the sample superpixel encoded features; uses the decoding network to decode the sample superpixel encoded features of the sample superpixel region, obtaining the sample superpixel decoded features; fuses the sample superpixel encoded features and the sample superpixel decoded features to obtain a fourth sample superpixel feature; and based on the fourth sample superpixel feature, determines the sample anomaly detection result for the sample superpixel region.

[0268] Optionally, the anomaly detection model includes an attention network and a classification network. The computer device uses the attention network to multiply the fourth sample superpixel feature with the mapping matrix to obtain the sample superpixel query feature, sample superpixel key feature, and sample superpixel value feature; then, using the attention network, it fuses the sample superpixel query feature, sample superpixel key feature, and sample superpixel value feature to obtain the fifth sample superpixel feature; finally, using the classification network, it transforms the fifth sample superpixel feature to obtain the sample anomaly detection result for the sample superpixel region.

[0269] The process of obtaining the sample anomaly detection result in step 703 is the same as the process of obtaining the anomaly detection result in steps 304-307 above, and will not be described in detail here.

[0270] 705. The computer equipment trains an anomaly detection model based on the anomaly detection label of each sample superpixel region and the sample anomaly detection result. The anomaly detection label indicates whether an anomaly truly exists in the sample superpixel region.

[0271] Anomaly detection labels reflect the true situation of whether there are anomalies in the superpixel regions of a sample, while the sample anomaly detection results are predictions based on the anomaly detection model, reflecting the predicted situation of whether there are anomalies in the superpixel regions of a sample. Therefore, the smaller the difference between the sample anomaly detection results and the anomaly detection labels, the more accurate the sample anomaly detection results are, which means the more accurate the resource recommendation model is. Thus, computer equipment trains the anomaly detection model based on anomaly detection labels and sample anomaly detection results, with the training objective of reducing the difference between the sample anomaly detection results and the anomaly detection labels, thereby improving the accuracy of the anomaly detection model.

[0272] In one possible implementation, the k different scales of the first sample superpixel features are obtained through the k different scales of the feature extraction network in the anomaly detection model. The k different first sample image features in step 701 are obtained through the k different scales of the feature extraction network in the superpixel segmentation model. The k different scales of the feature extraction network in the anomaly detection model are the same as the k different scales of the feature extraction network in the superpixel segmentation model.

[0273] Step 705 includes: training an anomaly detection model and a superpixel segmentation model based on anomaly detection labels of multiple sample superpixel regions, sample anomaly detection results, sample image segmentation labels, and sample segmentation results. The segmentation label refers to the result of superpixel segmentation of the sample image.

[0274] Since the feature extraction network in the anomaly detection model is the same as that in the superpixel segmentation model, and the superpixel regions obtained during anomaly detection are obtained through the superpixel segmentation model, the two related anomaly detection and superpixel segmentation tasks share the same feature extraction network. Therefore, to improve model performance, this embodiment utilizes the differences between the output of the anomaly detection model and the actual inspection results, as well as the differences between the output of the superpixel segmentation model and the actual segmentation results, to collaboratively train the anomaly detection model and the superpixel segmentation model.

[0275] Optionally, the computer device determines a first loss parameter based on the difference between the anomaly detection labels of multiple sample superpixel regions and the sample anomaly detection results. This first loss parameter is positively correlated with the difference and represents the loss of the anomaly detection task. The computer device also determines a second loss parameter based on the difference between the segmentation labels of sample images and the sample segmentation results. This second loss parameter is positively correlated with the difference and represents the loss of the superpixel segmentation task. Based on the first and second loss parameters, the computer device determines a target loss parameter and trains the anomaly detection model and the superpixel segmentation model based on the target loss parameter.

[0276] For example, the computer device weights the first loss parameter based on a first weight to obtain a third loss parameter, and weights the second loss parameter based on a second weight to obtain a fourth loss parameter. The third loss parameter and the fourth loss parameter are then added to obtain a first target loss parameter. An anomaly detection model is trained based on the first target loss parameter. The first weight is greater than the second weight.

[0277] For example, the computer device weights the first loss parameter based on the second weight to obtain the fifth loss parameter, and then weights the second loss parameter based on the first weight to obtain the sixth loss parameter. The fifth and sixth loss parameters are then added to obtain the second target loss parameter, and a superpixel segmentation model is trained based on the second target loss parameter. Here, the first weight is greater than the second weight.

[0278] In other words, when training an anomaly detection model, the proportion of the first loss parameter is greater than that of the second loss parameter, or the proportion of the loss parameter for the anomaly detection task is greater than that for the superpixel segmentation task, so that the anomaly detection model learns more features relevant to the anomaly detection task. Similarly, when training a superpixel segmentation model, the proportion of the second loss parameter is greater than that of the first loss parameter, or the proportion of the loss parameter for the superpixel segmentation task is greater than that for the anomaly detection task, so that the superpixel segmentation model learns more features relevant to the superpixel segmentation task.

[0279] For example, computer equipment uses the following formulas (18) and (19) to determine the first target loss parameter and the second target loss parameter.

[0280] L1=λLflower+(1-λ)Lboundary; formula (18)

[0281] L2=(1-λ)Lflower+λboundary; formula (19)

[0282] Where L1 represents the first objective loss parameter, L2 represents the second objective loss parameter, Lflower represents the first loss parameter, Lboundary represents the second loss parameter, λ represents the first weight, 1-λ represents the second weight, and λ is greater than 1-λ, and both λ and 1-λ are positive numbers.

[0283] In this implementation, the k feature extraction networks at different scales in the anomaly detection model are the same as those in the superpixel segmentation model. Sharing the same feature extraction networks simplifies the overall structure, ensures consistency in feature extraction, and enhances the synergistic effect of cross-model learning. Furthermore, co-training the anomaly detection model and the superpixel segmentation model enables multi-task learning, leveraging the correlation between different tasks to achieve mutual promotion and improve the generalization ability of both models.

[0284] The method provided in this application segmentes the image to be detected into multiple superpixel regions, and performs local feature extraction and fusion on a superpixel region basis to identify the probability of anomalies in each superpixel region. This reduces unnecessary global computation, lowers the computational complexity of the anomaly detection model, and improves the efficiency of anomaly detection. Furthermore, by extracting and fusing features from multiple superpixel regions at different scales, it is beneficial to capture features at different scales, ensuring the robustness and accuracy of the anomaly detection model. Moreover, detecting on a superpixel region basis can effectively identify local anomalies in the image, improving the precision of the anomaly detection model.

[0285] The image anomaly detection method provided in this application embodiment can be applied to any scenario where anomaly detection of displayed content is required.

[0286] For example, in the field of video games, the method provided in the embodiments of this application can be used to detect anomalies in game screens. Detailed procedures are described below. Figure 8 The example shown. Figure 8 This is a flowchart of another image anomaly detection method provided in the embodiments of this application, such as... Figure 8 As shown, the method includes the following steps:

[0287] 801. The terminal displays the game screen in the game client. The game client can be a 3D game client or a 2D game client; this embodiment of the application does not limit this.

[0288] 802. In response to the anomaly detection command for the game screen, the terminal takes a screenshot of the game screen to obtain the game screen image.

[0289] 803. The terminal sends an anomaly detection request carrying game screen images to the server.

[0290] 804. In response to an anomaly detection request, the server identifies multiple superpixel regions in the game screen image. The superpixel regions include multiple pixels in the game screen image.

[0291] 805. The server extracts k first superpixel features of different scales from each superpixel region, where k is an integer greater than 1.

[0292] 806. The server fuses the first superpixel features of k different scales in each superpixel region to obtain the superpixel fused features of each superpixel region;

[0293] 807. Based on the superpixel fusion features of each superpixel region, the server determines the anomaly detection result for each superpixel region. The anomaly detection result represents the probability that there is an anomaly in the superpixel region of the image.

[0294] 808. The server returns the anomaly detection result to the terminal.

[0295] 809. The terminal receives the anomaly detection result and displays the anomaly detection result.

[0296] This application addresses the problem of anomalies in game screens by proposing a game screen anomaly detection scheme based on multi-scale feature fusion and superpixel region segmentation. The scheme segments the game screen image of the game client into multiple superpixel regions, and performs local feature extraction and fusion on a per-superpixel-region basis to identify the probability of anomalies in each superpixel region. This reduces unnecessary global computation, lowers the computational complexity of the anomaly detection model, eliminates the need for manual judgment of the game screen of the game client, and improves the efficiency of anomaly detection.

[0297] Figure 9 This is a schematic diagram of the structure of an image anomaly detection device provided in an embodiment of this application. See also... Figure 9 The device includes:

[0298] The determination module 901 is used to determine multiple superpixel regions of an image, wherein the superpixel regions include multiple pixels in the image;

[0299] The feature extraction module 902 is used to extract k first superpixel features of different scales from each superpixel region, where k is an integer greater than 1;

[0300] The feature fusion module 903 is used to fuse the first superpixel features of k different scales of each superpixel region to obtain the superpixel fused features of each superpixel region.

[0301] The detection module 904 is used to determine the anomaly detection result of each superpixel region based on the superpixel fusion features of each superpixel region. The anomaly detection result represents the probability that there is an anomaly in the superpixel region of the image.

[0302] The image anomaly detection apparatus provided in this application identifies multiple superpixel regions of an image to be detected. It then performs local feature extraction and fusion on a per-superpixel-region basis to identify the probability of anomalies in each superpixel region. This reduces unnecessary global computation, lowers computational complexity, and eliminates the need for manual review, thus improving the efficiency of anomaly detection. Furthermore, by extracting and fusing features from multiple superpixel regions at different scales, it facilitates the capture of features at different scales, ensuring the robustness and accuracy of anomaly detection. Moreover, detecting on a per-superpixel-region basis effectively identifies local anomalies in the image, improving the precision of anomaly detection.

[0303] Optionally, module 901 is configured to:

[0304] Identify multiple cluster center points in the image;

[0305] Multiple clusters are obtained by clustering multiple pixels in an image based on multiple cluster centroids.

[0306] A region consisting of multiple pixels in any cluster is defined as a superpixel region.

[0307] Optionally, module 901 is configured to:

[0308] For any pixel, the feature distance between the pixel and multiple cluster centers is determined based on the color distance and spatial distance between the pixel and multiple cluster centers. The color distance refers to the distance between the color parameters of the pixel and the color parameters of the cluster centers, and the spatial distance refers to the distance between the position parameters of the pixel and the position parameters of the cluster centers.

[0309] Pixels are assigned to the cluster containing the cluster center point with the smallest feature distance to the pixel.

[0310] Optionally, the anomaly detection model includes k feature extraction networks of different scales, each feature extraction network including multiple feature extraction layers connected in sequence;

[0311] Feature extraction module 902 is used for:

[0312] The superpixel region is input into k feature extraction networks in the anomaly detection model to obtain the superpixel features output by the first feature extraction layer and the superpixel features output by the last feature extraction layer in the k feature extraction networks.

[0313] For each feature extraction network, the superpixel features output by the first feature extraction layer and the superpixel features output by the last feature extraction layer are fused to obtain k first superpixel features at different scales.

[0314] Optionally, the feature fusion module 903 is used for:

[0315] By fusing the first superpixel features of k different scales in the superpixel region, the second superpixel feature of the superpixel region is obtained;

[0316] Determine the adjacent superpixel regions of the superpixel region. The adjacent superpixel regions of the superpixel region refer to other superpixel regions that are adjacent to the superpixel region.

[0317] The second superpixel feature of the superpixel region and the second superpixel feature of the adjacent superpixel region are fused to obtain the superpixel fused pixel feature of the superpixel region.

[0318] Optionally, the feature fusion module 903 is used for:

[0319] The weights between the superpixel regions and their neighboring superpixel regions are determined based on the difference between the second superpixel features of the superpixel region and the second superpixel features of the adjacent superpixel regions. The weights are negatively correlated with the differences.

[0320] Based on the weights between the superpixel region and its adjacent superpixel regions, the second superpixel features of the adjacent superpixel regions are weighted to obtain the third superpixel features of the adjacent superpixel regions.

[0321] The superpixel fusion feature of the superpixel region is obtained by adding the second superpixel feature of the superpixel region to the third superpixel feature of the adjacent superpixel region.

[0322] Optionally, the anomaly detection model includes an encoding network and a decoding network; the detection module 904 is used for:

[0323] The superpixel fusion features of the superpixel region are encoded using an encoding network to obtain the superpixel encoded features;

[0324] By using a decoding network, the superpixel coding features of the superpixel region are decoded to obtain the superpixel decoding features;

[0325] By fusing superpixel encoding features and superpixel decoding features, a fourth superpixel feature is obtained;

[0326] Based on the fourth superpixel feature, the anomaly detection results of the superpixel region are determined.

[0327] Optionally, the encoding network includes m encoding layers connected in sequence, and the decoding network includes m decoding layers connected in sequence, where m is an integer greater than 1; the detection module 904 is used for:

[0328] The superpixel fusion features are input into the coding network to obtain the superpixel coding features output by m coding layers in the coding network;

[0329] By using a decoding network, the superpixel encoded features of the superpixel region are decoded to obtain the superpixel decoded features:

[0330] The superpixel encoded features output from the m-th encoding layer are input into the first decoding layer to obtain the superpixel decoded features output from the first decoding layer.

[0331] The superpixel encoded features output from the nth encoding layer and the superpixel decoded features output from the mnth decoding layer are input into the (m+1)nth decoding layer to obtain the superpixel decoded features output from the (m+1)nth decoding layer, where n is a positive integer less than m.

[0332] Optionally, the detection module 904 is used for:

[0333] The fourth superpixel feature is obtained by fusing the superpixel coding features output from the first coding layer, the superpixel coding features output from the m-th coding layer, the superpixel decoding features output from the first decoding layer, and the superpixel decoding features output from the m-th decoding layer.

[0334] Optionally, the anomaly detection model includes an attention network and a classification network; the detection module 904 is used for:

[0335] By using an attention network, the fourth superpixel feature is multiplied by the mapping matrix to obtain the superpixel query feature, superpixel key feature, and superpixel value feature;

[0336] By using an attention network, the superpixel query features, superpixel key features, and superpixel value features are fused to obtain the fifth superpixel feature;

[0337] By transforming the fifth superpixel feature through a classification network, the detection results of the superpixel region are obtained. The classification network is used to predict the probability of an anomaly.

[0338] Optionally, module 901 is configured to:

[0339] Extract k first image features at different scales from the image;

[0340] The image fusion features are obtained by fusing the first image features at k different scales of the image.

[0341] Based on the image fusion features, the image segmentation result is determined. The segmentation result refers to the result of superpixel segmentation of the image.

[0342] Optionally, the superpixel segmentation model includes k feature extraction networks of different scales, each feature extraction network including multiple feature extraction layers connected in sequence; the determination module 901 is used for:

[0343] The image is input into k feature extraction networks in the superpixel segmentation model to obtain the image features output by the first feature extraction layer and the image features output by the last feature extraction layer in the k feature extraction networks;

[0344] For each feature extraction network, the image features output by the first feature extraction layer and the image features output by the last feature extraction layer are fused to obtain k first image features at different scales.

[0345] Optionally, see Figure 10 The image anomaly detection device is implemented through an anomaly detection model. The device also includes a model training module 905, used for:

[0346] Identify multiple superpixel regions of the sample image, where each superpixel region includes multiple pixels in the sample image;

[0347] Using an anomaly detection model, k first sample superpixel features at different scales are extracted from each sample superpixel region;

[0348] By fusing the first superpixel features of k different scales of each sample superpixel region, the sample superpixel fused features of each sample superpixel region are obtained.

[0349] Using the anomaly detection model, based on the sample superpixel fusion features of each sample superpixel region, the sample anomaly detection result for each sample superpixel region is determined.

[0350] An anomaly detection model is trained based on the anomaly detection label of each sample superpixel region and the sample anomaly detection result. The anomaly detection label indicates whether an anomaly truly exists in the sample superpixel region.

[0351] Optionally, see Figure 10 Model training module 905 is used for:

[0352] Using a superpixel segmentation model, k features of the first sample image at different scales are extracted from the sample image.

[0353] The sample image fusion features are obtained by fusing the first sample image features at k different scales of the sample image.

[0354] The superpixel segmentation model determines the sample segmentation result of the sample image based on the sample image fusion features. The sample segmentation result refers to the result of superpixel segmentation of the sample image.

[0355] Optionally, see Figure 10The first sample superpixel features of k different scales are obtained through the feature extraction network of k different scales in the anomaly detection model. The first sample image features of k different scales are obtained through the feature extraction network of k different scales in the superpixel segmentation model. The feature extraction network of k different scales in the anomaly detection model is the same as the feature extraction network of k different scales in the superpixel segmentation model.

[0356] Model training module 905 is used for:

[0357] Based on the anomaly detection label of each sample superpixel region, the sample anomaly detection result, the sample image segmentation label, and the sample segmentation result, an anomaly detection model and a superpixel segmentation model are trained. The segmentation label refers to the result of superpixel segmentation of the sample image.

[0358] It should be noted that the image anomaly detection device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image anomaly detection device and the image anomaly detection method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0359] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed in the image anomaly detection method of the above embodiments.

[0360] Optionally, the computer device is provided as a terminal. Figure 11 A schematic diagram of the structure of a terminal 1100 provided in an exemplary embodiment of this application is shown.

[0361] Terminal 1100 includes a processor 1101 and a memory 1102.

[0362] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0363] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one computer program, which is used by the processor 1101 to implement the image anomaly detection method provided in the method embodiments of this application.

[0364] In some embodiments, the terminal 1100 may also optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. Optionally, the peripheral device includes at least one of: a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, and a power supply 1108.

[0365] Peripheral device interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1101 and memory 1102. In some embodiments, processor 1101, memory 1102 and peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1101, memory 1102 and peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0366] The radio frequency (RF) circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1104 can communicate with other devices via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1104 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0367] Display screen 1105 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1105 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1101 for processing. In this case, display screen 1105 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1105, disposed on the front panel of terminal 1100; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1100 or in a folded design; in still other embodiments, display screen 1105 may be a flexible display screen, disposed on a curved or folded surface of terminal 1100. Furthermore, display screen 1105 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0368] The camera assembly 1106 is used to acquire images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal 1100, and the rear-facing camera is disposed on the back of the terminal 1100. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash may be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0369] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1101 for processing, or input to the radio frequency circuit 1104 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0370] Power supply 1108 is used to power the various components in terminal 1100. Power supply 1108 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1108 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0371] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0372] Optionally, the computer device is provided as a server. Figure 12 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1200 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1201 and one or more memories 1202. The memories 1202 store at least one computer program, which is loaded and executed by the processor 1201 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0373] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the image anomaly detection method of the above embodiments.

[0374] This application also provides a computer program product, including a computer program loaded and executed by a processor to perform the operations performed by the image anomaly detection method of the above embodiments.

[0375] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0376] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. An image anomaly detection method, characterized in that, The method includes: Identify multiple superpixel regions of an image, wherein the superpixel regions include multiple pixels in the image; Extract k first superpixel features of different scales from each superpixel region, where k is an integer greater than 1; By fusing the k different scales of the first superpixel features of each superpixel region, a superpixel fused feature for each superpixel region is obtained; Based on the superpixel fusion features of each superpixel region, an anomaly detection result is determined for each superpixel region, wherein the anomaly detection result represents the probability that an anomaly exists in the superpixel region of the image.

2. The method according to claim 1, characterized in that, The determination of multiple superpixel regions of the image includes: Multiple cluster center points were identified in the image; Based on multiple clustering center points, multiple pixels in the image are clustered to obtain multiple clusters; The region formed by multiple pixels in any of the clusters is defined as a superpixel region.

3. The method according to claim 2, characterized in that, The process of clustering multiple pixels in the image based on multiple clustering center points to obtain multiple clusters includes: For any given pixel, the feature distance between the pixel and the multiple cluster centers is determined based on the color distance and spatial distance between the pixel and the multiple cluster centers, respectively. The color distance refers to the distance between the color parameters of the pixel and the color parameters of the cluster centers, and the spatial distance refers to the distance between the position parameters of the pixel and the position parameters of the cluster centers. The pixel is assigned to the cluster containing the cluster center point with the smallest feature distance to the pixel.

4. The method according to claim 1, characterized in that, The anomaly detection model includes k feature extraction networks of different scales, and each feature extraction network includes multiple feature extraction layers connected in sequence; The step of extracting k first superpixel features of different scales from each of the superpixel regions includes: The superpixel regions are respectively input into k feature extraction networks in the anomaly detection model to obtain the superpixel features output by the first feature extraction layer and the superpixel features output by the last feature extraction layer in the k feature extraction networks; For each of the feature extraction networks, the superpixel features output by the first feature extraction layer and the superpixel features output by the last feature extraction layer are fused to obtain k first superpixel features of different scales.

5. The method according to claim 1, characterized in that, The process of fusing the k different scales of the first superpixel features of each superpixel region to obtain the superpixel fused feature of each superpixel region includes: By fusing k first superpixel features of different scales in the superpixel region, a second superpixel feature of the superpixel region is obtained; Determine the adjacent superpixel regions of the superpixel region, where the adjacent superpixel regions refer to other superpixel regions adjacent to the superpixel region; The second superpixel feature of the superpixel region and the second superpixel feature of the adjacent superpixel region are fused to obtain the superpixel fused pixel feature of the superpixel region.

6. The method according to claim 5, characterized in that, The process of fusing the second superpixel feature of the superpixel region and the second superpixel feature of the adjacent superpixel region to obtain the superpixel fused feature of the superpixel region includes: Based on the difference between the second superpixel feature of the superpixel region and the second superpixel feature of the adjacent superpixel region, a weight is determined between the superpixel region and the adjacent superpixel region, and the weight is negatively correlated with the difference. Based on the weights between the superpixel region and the adjacent superpixel region, the second superpixel features of the adjacent superpixel region are weighted to obtain the third superpixel features of the adjacent superpixel region. The second superpixel feature of the superpixel region is added to the third superpixel feature of the adjacent superpixel region to obtain the superpixel fusion feature of the superpixel region.

7. The method according to claim 1, characterized in that, The anomaly detection model includes an encoding network and a decoding network; the determination of the anomaly detection result for each superpixel region based on the superpixel fusion features of each superpixel region includes: The superpixel fusion features of the superpixel region are encoded through the coding network to obtain superpixel coded features; The superpixel coding features of the superpixel region are decoded using the decoding network to obtain the superpixel decoding features; By fusing the superpixel encoding features and the superpixel decoding features, a fourth superpixel feature is obtained; Based on the fourth superpixel feature, the anomaly detection result of the superpixel region is determined.

8. The method according to claim 7, characterized in that, The encoding network comprises m sequentially connected encoding layers, and the decoding network comprises m sequentially connected decoding layers, where m is an integer greater than 1; the process of encoding the superpixel fusion features of the superpixel region through the encoding network to obtain superpixel encoded features includes: The superpixel fusion features are input into the coding network to obtain the superpixel coding features output by m coding layers in the coding network; The superpixel coding features of the superpixel region are decoded through the decoding network to obtain the superpixel decoding features: The superpixel encoded features output from the m-th encoding layer are input into the first decoding layer to obtain the superpixel decoded features output from the first decoding layer. The superpixel encoded features output from the nth encoding layer and the superpixel decoded features output from the mnth decoding layer are input into the (m+1)nth decoding layer to obtain the superpixel decoded features output from the (m+1)nth decoding layer, where n is a positive integer less than m.

9. The method according to claim 8, characterized in that, The fusion of the superpixel encoding features and the superpixel decoding features to obtain the fourth superpixel feature includes: The fourth superpixel feature is obtained by fusing the superpixel coding features output from the first coding layer, the superpixel coding features output from the m-th coding layer, the superpixel decoding features output from the first decoding layer, and the superpixel decoding features output from the m-th decoding layer.

10. The method according to claim 7, characterized in that, The anomaly detection model includes an attention network and a classification network; determining the anomaly detection result of the superpixel region based on the fourth superpixel feature includes: By multiplying the fourth superpixel feature with the mapping matrix through the attention network, we obtain the superpixel query feature, the superpixel key feature, and the superpixel value feature. The fifth superpixel feature is obtained by fusing the superpixel query feature, the superpixel key feature, and the superpixel value feature through the attention network. The classification network is used to transform the features of the fifth superpixel to obtain the anomaly detection result of the superpixel region. The classification network is used to predict the probability of an anomaly.

11. The method according to claim 1, characterized in that, The determination of multiple superpixel regions of the image includes: Extract k first image features at different scales from the image; By fusing k different scales of the first image features of the image, the image fusion features of the image are obtained; Based on the image fusion features of the image, the segmentation result of the image is determined, wherein the segmentation result refers to the result of superpixel segmentation of the image.

12. The method according to claim 11, characterized in that, The superpixel segmentation model includes k feature extraction networks at different scales, each of which includes multiple feature extraction layers connected in sequence; the step of extracting k first image features at different scales from the image includes: The image is input into k feature extraction networks in the superpixel segmentation model to obtain the image features output by the first feature extraction layer and the image features output by the last feature extraction layer in the k feature extraction networks. For each of the feature extraction networks, the image features output by the first feature extraction layer and the image features output by the last feature extraction layer are fused to obtain k first image features at different scales.

13. The method according to claim 1, characterized in that, The image anomaly detection method is implemented through an anomaly detection model, and the training process of the anomaly detection model includes: Determine multiple superpixel regions of a sample image, wherein the superpixel regions include multiple pixels in the sample image; Using the anomaly detection model, k first sample superpixel features of different scales are extracted from each sample superpixel region; By fusing the k different scales of the first sample superpixel features of each sample superpixel region, the sample superpixel fusion feature of each sample superpixel region is obtained; Using the anomaly detection model, based on the sample superpixel fusion features of each sample superpixel region, the sample anomaly detection result for each sample superpixel region is determined respectively; The anomaly detection model is trained based on the anomaly detection label of each superpixel region of the sample and the anomaly detection result of the sample. The anomaly detection label indicates whether an anomaly truly exists in the superpixel region of the sample.

14. The method according to claim 13, characterized in that, The determination of multiple superpixel regions of the sample image includes: Using a superpixel segmentation model, k first sample image features at different scales are extracted from the sample image. The sample image fusion feature of the sample image is obtained by fusing the first sample image features at k different scales of the sample image; Based on the sample image fusion features of the sample image, the sample segmentation result of the sample image is determined by the superpixel segmentation model. The sample segmentation result refers to the result of superpixel segmentation of the sample image.

15. The method according to claim 14, characterized in that, The k different scales of the first sample superpixel features are obtained through the k different scales of the feature extraction network in the anomaly detection model. The k different first sample image features are obtained through the k different scales of the feature extraction network in the superpixel segmentation model. The k different scales of the feature extraction network in the anomaly detection model are the same as the k different scales of the feature extraction network in the superpixel segmentation model. The step of training the anomaly detection model based on the anomaly detection label of each superpixel region of the sample and the anomaly detection result of the sample includes: Based on the anomaly detection label of each superpixel region of the sample, the anomaly detection result of the sample, the segmentation label of the sample image, and the sample segmentation result, the anomaly detection model and the superpixel segmentation model are trained, where the segmentation label refers to the result of superpixel segmentation of the sample image.

16. An image anomaly detection device, characterized in that, The device includes: A determination module is used to determine multiple superpixel regions of an image, wherein the superpixel regions include multiple pixels in the image; The feature extraction module is used to extract k first superpixel features of different scales from each superpixel region, where k is an integer greater than 1; The feature fusion module is used to fuse the k different scales of the first superpixel features of each superpixel region to obtain the superpixel fused feature of each superpixel region. The detection module is used to determine the anomaly detection result of each superpixel region based on the superpixel fusion feature of each superpixel region, wherein the anomaly detection result represents the probability that there is an anomaly in the superpixel region of the image.

17. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to perform the operations of the image anomaly detection method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to perform the operations of the image anomaly detection method as described in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, The computer program is loaded and executed by a processor to perform the operations of the image anomaly detection method as described in any one of claims 1 to 15.

Images