Abnormal image detection method, generation method, device and computer equipment

By using a backbone network and a feature fusion network with pointwise convolution and channelwise convolution in the video abnormal image detection model, a lightweight model is constructed, which solves the problem of real-time detection of multiple types of screen distortion on resource-limited devices in the existing technology and achieves efficient abnormal image detection.

CN118135249BActive Publication Date: 2026-11-10WUHAN UNIV +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202211536562.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-02
Publication Date
2026-11-10
Estimated Expiration
2042-12-02

AI Technical Summary

Technical Problem

Existing video anomaly detection methods are difficult to implement in real time on resource-constrained mobile and embedded devices, and the types of detection are limited, failing to meet the diverse needs for detecting screen distortion.

Method used

An anomaly image detection model employing a backbone network and a feature fusion network is proposed. Feature extraction is performed using pointwise convolution and channelwise convolution to construct a lightweight detection model, reducing computational resource consumption.

Benefits of technology

It enables real-time detection of various types of abnormal images on edge devices, reduces algorithm complexity and computational load, and improves the applicability of the detection model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118135249B_ABST
    Figure CN118135249B_ABST
Patent Text Reader

Abstract

This application relates to a method, generation method, apparatus, and computer device for detecting abnormal images. The method acquires an image to be detected and inputs it into a preset abnormal image detection model for anomaly detection, obtaining a detection result. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image. Compared to methods using ordinary convolution kernels for feature extraction, the backbone network composed of pointwise and channelwise convolutions in this method significantly reduces computational load while extracting features, making the entire abnormal image detection model a lightweight detection model. This reduces the resource consumption of the abnormal image detection model, allowing it to be deployed to edge devices for real-time detection and improving its applicability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method, method, apparatus, and computer device for detecting and generating abnormal images. Background Technology

[0002] With the development of internet and multimedia technologies, video playback is affected by various factors such as network transmission performance limitations, encoding / decoding defects, or other interference. During transmission and storage, some frames may be corrupted, considered abnormal frames, such as stripes, mosaics, tearing, motion residuals, green screens, and other screen distortion phenomena. Therefore, detecting abnormal images in videos is crucial for improving the user's video playback experience.

[0003] Currently, methods for detecting distorted images include: single gradient value method of video frames, gray-level gradient value method of video frame edge images, edge pixel method of image sub-regions, color space transformation method, etc. However, these methods can detect only a limited number of distorted image types. Therefore, methods have emerged that train deep learning neural networks as classifiers to achieve distorted image detection.

[0004] However, current neural network algorithms used to detect screen distortion are quite complex, making it difficult to detect screen distortion in real time on edge devices such as mobile devices and embedded devices with limited resources. Summary of the Invention

[0005] Therefore, it is necessary to provide a method for detecting and generating abnormal images that can reduce resource consumption, as well as corresponding devices, computer equipment, storage media, and computer program products, to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for detecting abnormal images. The method includes:

[0007] Acquire the image to be detected;

[0008] The image to be detected is input into a preset abnormal image detection model for anomaly detection, and the detection result is obtained. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image.

[0009] Secondly, this application also provides a method for generating abnormal images, the method comprising:

[0010] The target data packet corresponding to the first image in the initial image set is processed to obtain an image with an abnormal shape;

[0011] The pixel values ​​of the second image in the initial image set are processed to obtain a color anomaly image;

[0012] The abnormal shape image and the abnormal color image are combined to form a sample image set; the sample image set is used to train an abnormal image detection model as described in the first aspect.

[0013] Thirdly, this application also provides an apparatus for detecting abnormal images. The apparatus includes:

[0014] The acquisition module is used to acquire the image to be detected;

[0015] The detection module is used to input the image to be detected into a preset abnormal image detection model for abnormal detection and obtain the detection result. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image.

[0016] Fourthly, this application also provides an apparatus for generating abnormal images. The apparatus includes:

[0017] The first processing module is used to process the target data packet corresponding to the first image in the initial image set to obtain an image with an abnormal shape;

[0018] The second processing module is used to process the pixel values ​​of the second image in the initial image set to obtain a color anomaly image;

[0019] A construction module is used to assemble the shape-abnormal image and the color-abnormal image into a sample image set; the sample image set is used to train an abnormal image detection model.

[0020] Fifthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0021] Acquire the image to be detected;

[0022] The image to be detected is input into a preset abnormal image detection model for anomaly detection, and the detection result is obtained. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image.

[0023] Sixthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0024] The target data packet corresponding to the first image in the initial image set is processed to obtain an image with an abnormal shape;

[0025] The pixel values ​​of the second image in the initial image set are processed to obtain a color anomaly image;

[0026] The abnormal shape image and the abnormal color image are combined into a sample image set; the sample image set is used to train an abnormal image detection model.

[0027] Seventhly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0028] Acquire the image to be detected;

[0029] The image to be detected is input into a preset abnormal image detection model for anomaly detection, and the detection result is obtained. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image.

[0030] Eighthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0031] The target data packet corresponding to the first image in the initial image set is processed to obtain an image with an abnormal shape;

[0032] The pixel values ​​of the second image in the initial image set are processed to obtain a color anomaly image;

[0033] The abnormal shape image and the abnormal color image are combined into a sample image set; the sample image set is used to train an abnormal image detection model.

[0034] Ninthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0035] Acquire the image to be detected;

[0036] The image to be detected is input into a preset abnormal image detection model for anomaly detection, and the detection result is obtained. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image.

[0037] Tenthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0038] The target data packet corresponding to the first image in the initial image set is processed to obtain an image with an abnormal shape;

[0039] The pixel values ​​of the second image in the initial image set are processed to obtain a color anomaly image;

[0040] The abnormal shape image and the abnormal color image are combined into a sample image set; the sample image set is used to train an abnormal image detection model.

[0041] The abnormal image detection method, apparatus, computer equipment, storage medium, and computer program product provided in this application acquire an image to be detected and input it into a preset abnormal image detection model for anomaly detection to obtain a detection result. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image. Compared with the method of using ordinary convolution kernels to extract features from the image, pointwise convolution and channelwise convolution are lightweight and can greatly reduce the algorithm complexity. Therefore, the backbone network composed of pointwise convolution and channelwise convolution in this method can greatly reduce the amount of computation while extracting features, making the entire abnormal image detection model a lightweight detection network. This reduces the resource consumption of the abnormal image detection model, allowing it to be deployed to edge devices for real-time detection and improving the applicability of the abnormal image detection model. Attached Figure Description

[0042] Figure 1 This is an internal structural diagram of a computer device in one embodiment;

[0043] Figure 2 This is a flowchart illustrating an abnormal image detection method in one embodiment;

[0044] Figure 2A This is a schematic diagram of an abnormal image detection model in one embodiment;

[0045] Figure 3 This is a schematic diagram of the architecture of a lightweight neural network module in one embodiment;

[0046] Figure 4 This is a schematic diagram of the architecture of a lightweight neural network module in another embodiment;

[0047] Figure 5 This is a schematic diagram of the architecture of a lightweight neural network module in another embodiment;

[0048] Figure 6 This is a schematic diagram of the architecture of a lightweight neural network module in another embodiment;

[0049] Figure 7 This is a schematic diagram of the backbone network architecture in one embodiment;

[0050] Figure 8 This is a flowchart illustrating an abnormal image detection method in another embodiment;

[0051] Figure 9 This is a schematic diagram of the feature fusion network architecture in one embodiment;

[0052] Figure 10 This is a schematic diagram of the feature fusion network architecture in another embodiment;

[0053] Figure 11 This is a schematic diagram of the feature fusion network architecture in another embodiment;

[0054] Figure 12 This is a schematic diagram of the feature fusion network architecture in another embodiment;

[0055] Figure 13 A flowchart illustrating a method for training an anomaly image detection model in one embodiment;

[0056] Figure 14 This is a schematic diagram of the architecture of the initial neural network in one embodiment;

[0057] Figure 15 A flowchart illustrating a method for training an anomaly image detection model according to another embodiment;

[0058] Figure 16 This is a flowchart illustrating a method for generating abnormal images in one embodiment;

[0059] Figure 17 for Figure 16 A flowchart illustrating one implementation of S1101;

[0060] Figure 18 for Figure 16 A flowchart illustrating one implementation of S1102;

[0061] Figure 19 This is a flowchart illustrating a method for determining preset weights in one embodiment;

[0062] Figure 20 for Figure 18 A flowchart illustrating one implementation of S3302;

[0063] Figure 21 This is a flowchart illustrating a method for generating abnormal images in another embodiment;

[0064] Figure 22 This is a schematic diagram of an abnormal image in one embodiment;

[0065] Figure 23 This is a structural block diagram of an abnormal image detection device in one embodiment;

[0066] Figure 24 This is a structural block diagram of an apparatus for generating abnormal images in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] In abnormal image detection methods, such as the detection of distorted images, the following methods are currently mainly used. The first method is the gradient uniformity value method for video frames. This method specifically includes: performing edge detection on the video frames to determine the edge detection map; smoothing the video frames and performing gradient detection to determine the gradient map; determining the gradient uniformity value corresponding to the video frame based on the edge detection map and the gradient map; and determining the distorted area included in the video frame when the gradient uniformity value is less than a preset threshold. However, this method can detect a limited number of distorted image types, and has a high rate of false positives and false negatives.

[0069] The second method is based on the grayscale gradient values ​​of video frame edge images. This method specifically includes: acquiring the target image to be processed in the currently playing target video and calculating the corresponding edge image; wherein the pixels in the edge image possess grayscale gradient values; dividing the edge image into a specified number of sub-blocks, and selecting candidate sub-blocks from the specified number of sub-blocks based on the grayscale gradient values ​​of the pixels in the sub-blocks; detecting the grayscale gradient values ​​of pixels on the boundaries of the candidate sub-blocks, and determining whether the candidate sub-blocks are distorted sub-blocks based on the detection results; counting the total number of distorted sub-blocks in the edge image, and if the total number is greater than a specified threshold, determining that the target image is a distorted image. However, this method can detect a limited number of distorted image types, has many missed detections, and has a long computation time, making it unsuitable for deployment on edge devices for real-time detection.

[0070] The third method is the edge pixel method for image sub-regions. This method specifically includes: acquiring the edge image of the image to be detected and dividing the edge image into multiple sub-regions; obtaining the edge percentage of each edge pixel in each sub-region relative to all pixels, resulting in a percentage set, which includes the edge percentages corresponding to multiple sub-regions; determining the average value and fluctuation feature value of the multiple edge percentages in the percentage set, where the fluctuation feature value represents the fluctuation of the multiple edge percentages relative to the average value; and determining whether the image to be detected is a distorted image based on the average value and fluctuation feature value. However, this method can detect a limited number of distorted image types, has many missed detections, and is computationally time-consuming, making it unsuitable for deployment on edge devices for real-time detection.

[0071] The fourth method is the color space transformation method. This method specifically includes: acquiring the original video data; acquiring the image to be detected after rendering by the graphics processor; performing a color space transformation on the image to be detected to obtain the transformed data; and detecting whether there is screen distortion based on the transformed data and the original data. However, this method is only for detecting screen distortion caused by color abnormalities or snow-like screen abnormalities, and it requires comparison of both original data and real-time data for detection. It cannot meet the blind detection of screen distortion with only real-time data.

[0072] The fifth method is a video mosaic detection method based on clustering and template matching. It uses edge detection to detect image edges, then clustering to classify pixels, and finally template matching to detect mosaic blocks. However, this method only detects rectangular mosaic anomalies and cannot detect other issues such as stripes, tears, or color anomalies. Furthermore, the computation time is too long to meet the real-time detection requirements of edge devices.

[0073] The sixth method is deep learning or hybrid deep learning methods. Reconstruction error algorithms in deep learning are widely used in video anomaly detection. However, the current mainstream neural network models used in this method are highly complex and require significant computational resources, making it difficult to detect anomaly images in real-time on resource-constrained edge devices such as mobile devices and embedded devices. Further research is needed to reduce model complexity and improve detection speed.

[0074] To address the aforementioned issues, this application proposes an abnormal image detection method. The abnormal image detection model used requires minimal computational resources and can detect various types of abnormal images. The following embodiments describe this method in detail.

[0075] The abnormal image detection method provided in this application embodiment can be applied to, for example, Figure 1 The computer device shown is a terminal, and its internal structure diagram can be as follows. Figure 1As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for detecting abnormal images. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0076] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0077] In one embodiment, such as Figure 2 As shown, a method for detecting abnormal images is provided. Figure 1 The computer device in the process is the executing entity, and the method includes the following steps:

[0078] S101, acquire the image to be detected.

[0079] The image to be detected may or may not contain anomalous information. It may contain only one type of anomalous information or multiple types. For example, the image may contain one or more anomalous information such as stripes, mosaic, tearing, motion blur, or green screen.

[0080] In this embodiment, the computer device can obtain the image to be detected by downloading pictures, directly obtaining the image corresponding to the video as the image to be detected, or generating an image containing abnormal information as the image to be detected.

[0081] S102, input the image to be detected into the preset abnormal image detection model for abnormal detection, and obtain the detection result.

[0082] The abnormal image detection model includes a backbone network and a feature fusion network for feature fusion and analysis (see [link]). Figure 2A The abnormal image detection model shown is illustrated. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image. For example, the backbone network may include pointwise convolutional layers composed of pointwise convolutional kernels and channelwise convolutional layers composed of channelwise convolutional kernels. The feature fusion network can be specifically a neural network used to fuse and analyze the feature maps output by the backbone network to obtain the detection result. Optionally, this feature fusion network can be a pyramid network. The abnormal image detection model can be pre-trained by a computer device based on a set of sample images, which contains various types of abnormal images.

[0083] The detection result can indicate whether the image to be detected is abnormal, or it can indicate the type of abnormal information contained in the image. For example, the detection result may indicate that the image contains stripe-type abnormal information, or that the image contains mosaic-type abnormal information. Optionally, the detection result may also include both the result indicating whether the image is abnormal and the result indicating the type of abnormal information contained in the image.

[0084] In this embodiment, the computer device can pre-construct an initial backbone network and an initial feature fusion network based on a neural network architecture. These two networks then constitute an initial detection model. This initial detection model is further trained based on a sample image set containing various types of anomalous images, thus obtaining an anomaly image detection model. When performing anomaly detection on an image to be detected, the image can be input into the pre-trained anomaly image detection model to determine whether the image is an anomalous image or what type of anomalous information it contains. It should be noted that when pre-constructing the backbone network, since the backbone network uses pointwise convolution and channelwise convolution to extract features from the image, multiple cascaded pointwise convolutional layers and channelwise convolutional layers can be designed in the backbone network to achieve feature extraction.

[0085] The aforementioned abnormal image detection method acquires the image to be detected and inputs it into a preset abnormal image detection model for anomaly detection, obtaining the detection result. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image. Compared to methods using ordinary convolution kernels for feature extraction, pointwise and channelwise convolution are lightweight, significantly reducing algorithm complexity. Therefore, the backbone network composed of pointwise and channelwise convolution in this method can greatly reduce computation while extracting features, making the entire abnormal image detection model a lightweight detection model. This reduces the resource consumption of the abnormal image detection model, allowing it to be deployed to edge devices for real-time detection and improving its applicability.

[0086] In one embodiment, a backbone network is provided, comprising multiple cascaded lightweight neural network modules, each of which employs pointwise convolution and channelwise convolution to feature the image. Optionally, the lightweight neural network module may consist of multiple cascaded pointwise convolutional layers and channelwise convolutional layers; alternatively, the lightweight neural network module may also consist of multiple alternating cascaded pointwise convolutional layers and channelwise convolutional layers.

[0087] In channel-wise convolution, the number of convolution kernels is the same as the number of channels in the input feature map, and each kernel performs convolution operation with only one channel of the feature map. Channel-wise convolution produces an output feature map with the same number of channels as the input. Pointwise convolution uses a 1×1 kernel to perform convolution operation with the input feature map, and finally generates an output feature map with the same number of channels as the number of kernels.

[0088] For example, for the input feature map, using H to represent height, W to represent width, and Nch to represent the number of channels, the size of the input feature map is:

[0089] 1×N ch ×H×W;

[0090] For a regular convolution kernel, where F represents the kernel size and N represents the number of kernels, its convolution dimension is:

[0091] F×F×N ch ×N;

[0092] For channel-wise convolution, the convolution size is:

[0093] F×F×1×N ch ;

[0094] For pointwise convolution, the convolution size is:

[0095] 1×1×N ch ×N;

[0096] The total computational cost of ordinary convolution is:

[0097] Compute(C) = H × W × F × F × N ch ×N;

[0098] The computational cost of channel-wise convolution is:

[0099] Compute(DC) = H × W × F × F × N ch ;

[0100] The computational cost of pointwise convolution is:

[0101] Compute (PC) = H × W × N ch ×N;

[0102] Therefore, the total computational cost of the lightweight neural network module composed of channel-wise convolution and point-wise convolution is:

[0103] Compute(DPC)=Compute(PC)+Compute(DC);

[0104] At the same time, we can conclude that:

[0105] Compute (DPC) <Compute(C);

[0106] It is noted that the computational cost required by using channel-wise convolution and point-wise convolution is much less than that required by ordinary convolution. Therefore, compared with using ordinary convolution to construct a neural network module, the lightweight neural network module proposed in this embodiment can greatly reduce the computational cost while extracting features.

[0107] Optionally, a specific architecture for a lightweight neural network module is provided, such as... Figure 3 As shown, the lightweight neural network module 10 includes: a first branch 110, a second branch 120, and a channel splicing layer 130; wherein, the first branch 110 is used to perform pointwise convolution and channelwise convolution on the image to obtain the first branch feature map;

[0108] The second branch 120 is used to perform global average pooling and channel-wise convolution on the image to obtain the second branch feature map; the channel stitching layer 130 is used to stitch the first branch feature map and the second branch feature map together to obtain a stitched feature map. It should be noted that if the sizes of the first branch feature map and the second branch feature map are inconsistent, before stitching the first branch feature map and the second branch feature map together, the second branch 120 is also used to resize the second branch feature map to obtain a processed feature map; the size of the processed feature map is consistent with the size of the first branch feature map; correspondingly, the channel stitching layer 130 is used to stitch the first branch feature map and the resized second branch feature map together to obtain a stitched feature map.

[0109] Optionally, a specific architecture for a lightweight neural network module is also provided, such as Figure 4 As shown, the lightweight neural network module 10 also includes a channel shuffling layer 140, which is used to shuffle the spliced ​​feature map by channels.

[0110] Optionally, the first branch 110 can be composed of one or more cascaded pointwise convolutional layers and channelwise convolutional layers, or can be composed of one or more cascaded pointwise convolutional layers and channelwise convolutional layers alternately, used to perform pointwise convolution and channelwise convolution on the input image, and output the first branch feature map; this embodiment provides a specific architecture of the first branch, see [link to documentation]. Figure 6 The architecture of the lightweight neural network module shown, namely the first branch 110, includes: a first pointwise convolutional layer 1101, a first channelwise convolutional layer 1102, and a second pointwise convolutional layer 1103 connected in sequence.

[0111] Specifically, the first pointwise convolutional layer 1101 is used to perform pointwise convolution on the input image and output a first intermediate feature map. Then, the first intermediate feature map is normalized (BN) and activated (ReLU). The processed first intermediate feature map is then input into the first channelwise convolutional layer 1102 for channelwise convolution and outputs a second intermediate feature map. Then, the second intermediate feature map is normalized (BN) and input into the second pointwise convolutional layer 1103 for pointwise convolution and outputs a third intermediate feature map. After normalization (BN) and activation (ReLU) processing, the first branch feature map is obtained.

[0112] The workflow of the first branch is illustrated by example. (Refer to...) Figure 6 The lightweight neural network module shown assumes the input image size is:

[0113] B×N ch ×H×W;

[0114] Where B represents Batch Size, N ch H represents the number of channels in the input image, H represents the height of the input image, and W represents the width of the input image.

[0115] like Figure 6 As shown, the first branch 110 consists of a pointwise convolutional layer, a depthwise convolutional layer, a normalization layer (BN layer), and an activation layer (ReLU layer) (the BN and ReLU layers are not explicitly shown in the figure, but are indicated by symbols). The stride of the depthwise convolutional layer is S. The input image passes through the first pointwise convolutional layer, BN layer, ReLU layer, first depthwise convolutional layer, BN layer, second pointwise convolutional layer, BN layer, and ReLU layer in sequence. The number of kernels in the second pointwise convolutional layer is Kch. The size of the first branch feature map output by the first branch is:

[0116]

[0117] The second branch 120 can be composed of one or more cascaded global average pooling layers 1201 and channel-wise convolutional layers 1202, used to perform global average pooling and channel-wise convolution on the image, outputting a second branch feature map. Specifically, the global pooling layer 1201 can calculate the average value for each channel of the input image. This embodiment provides a specific architecture of the second branch 120, see [link to documentation]. Figure 6 The lightweight neural network module shown, namely the second branch 120, includes a global pooling layer 1201 and a third pointwise convolutional layer 1202 connected in sequence.

[0118] Specifically, the global pooling layer 1201 is used to perform global average pooling on the input image and output the fourth intermediate feature map. The fourth intermediate feature map is then input into the third pointwise convolutional layer 1202 for pointwise convolution to obtain the fifth intermediate feature map. After normalization (BN) processing, the second branch feature map is obtained.

[0119] This example illustrates the workflow of the second branch. (Refer to...) Figure 6 The lightweight neural network module shown sequentially passes the input image through a global pooling layer, a third pointwise convolutional layer, and a batch normalization (BN) layer (not explicitly shown in the diagram, but labeled). After averaging through the global pooling layer, the size of the resulting fourth intermediate feature map is:

[0120] B×N ch ×1×1;

[0121] The output of the global pooling layer is fed into the third pointwise convolutional layer (Depthwise Conv layer) for convolution operations, resulting in the following final output size:

[0122] B×N ch ×1×1;

[0123] Next, adopt Figure 3 The channel stitching layer 130 in the middle can combine and stitch together feature maps with equal batch size, equal width and height, and different number of channels according to the channel dimension.

[0124] Suppose that the dimensions of the multiple input images are as follows:

[0125] {B×N1 ch ×H×W,B×N2 ch ×H×W,B×N3 ch ×H×W,......};

[0126] The output feature map can be obtained by concatenating channels, with the following dimensions:

[0127] B×(N1 ch +N2 ch +N3 ch +.....)×H×W;

[0128] Next, adopt Figure 4 The channel shuffling layer 140 in the code reorganizes the input feature map along the channel dimension to facilitate information exchange between different channels. Specifically, for the input feature map, if there are N channels... ch Divide the channels into g groups, while simultaneously satisfying N ch Since g = 0, there are g groups {0, 1, 2, ..., g-1}, and each group has N elements. ch / g channels.

[0129] For example, for the y-th channel in group x, it is reassigned to the x-th channel in group y. For a size of B×N... ch The input feature map of ×H×W is implemented as follows:

[0130] The second dimension N of the input feature map ch Decomposed into two dimensions, the feature map is obtained:

[0131]

[0132] Transpose the feature matrix after decomposition to obtain the feature map with the following dimensions:

[0133]

[0134] The second and third dimension feature matrices are reduced to one dimension, and the channels are rearranged to obtain the final output feature map with the following dimensions:

[0135] B×N ch ×H×W;

[0136] Optionally, another specific architecture for a lightweight neural network module is provided, such as Figure 5 As shown, the lightweight neural network module 10 further includes: a third branch 150 and a branch adder 160; the third branch 150 includes one or more second channel-wise convolutions 1501. The third branch 150 is used to perform channel-wise convolution on the image to obtain a third branch feature map. Correspondingly, the second branch 120 is also used to perform size processing on the second branch feature map to obtain a processed feature map; the size of the processed feature map is consistent with the size of the third branch feature map. The branch adder 150 is used to fuse (sum) the processed feature map and the third branch feature map to obtain a fourth branch feature map. Correspondingly, the channel concatenation layer is used to concatenate the first branch feature map and the fourth branch feature map to obtain a concatenated feature map.

[0137] The workflow of the third branch is illustrated below. (Refer to...) Figure 6 The lightweight neural network module shown has a third branch 150 containing a second depthwise convolutional layer, a batch normalization (BN) layer, and a ReLU activation layer (the BN and ReLU activation layers are not shown in the figure, but are labeled). The main function of the third branch 150 is to provide residual structure to the module to preserve feature information and mitigate gradient vanishing or exploding. The input image passes through the second depthwise convolutional layer and the BN layer sequentially. The size, number, and stride of the second depthwise convolutional kernel are the same as the first depthwise convolutional kernel in the first branch 110, ensuring that the width and height of the output feature map are the same as those of the first branch. The output of the third branch is:

[0138]

[0139] The outputs of the second branch 120 and the third branch 150 are added together. Since the feature map sizes of the two branches are mismatched, the scalar values ​​of each channel in the output of the second branch 120 are first broadcast to match the size of its feature map with that of the third branch 150. Then, the corresponding scalar values ​​are added within the matrix to obtain the output feature map, with the following size:

[0140]

[0141] The output feature map is then fed into a ReLU activation layer, and the resulting output feature map is concatenated with the feature map output from the first branch to obtain the output feature map, the size of which is:

[0142]

[0143] Finally, the output obtained by concatenating the channels can be shuffled while keeping the output size unchanged, thus obtaining the final output of the lightweight neural network module.

[0144] According to the foregoing Figures 3-6 The lightweight neural network module described in the embodiments shows that its first branch consists of pointwise convolutional layers and channelwise convolutional layers, which can reduce computational load to a certain extent, thereby reducing the resources required by the lightweight neural network module. Similarly, the second branch consists of global pooling layers and pointwise convolutional layers, which can also reduce computational load to a certain extent, thus reducing the resources required by the lightweight neural network module. Therefore, when the lightweight neural network module provided in the above embodiments is applied to an anomaly image detection model, it can greatly reduce computational load, thereby improving the detection speed of the anomaly image detection model.

[0145] In one embodiment, a backbone network architecture is also provided, wherein the backbone network, in addition to including the lightweight neural network module 10, also includes convolutional kernels 20 and pooling layers 30 (see Figure 10). Figure 7 The backbone network architecture shown can specifically include multiple convolutional kernels 20, pooling layers 30, and multiple lightweight neural network modules 10 connected in sequence. The process of using this backbone network in conjunction with a feature fusion network to achieve abnormal image detection is as follows: Figure 8 As shown, it includes:

[0146] S201, multiple convolutional kernels are used to extract features from the image to be detected, resulting in the first feature map.

[0147] S202, the first feature map is pooled using a pooling layer to obtain the second feature map.

[0148] S203 uses multiple lightweight neural network modules to extract features from the second feature map, resulting in multiple third feature maps.

[0149] S204. Multiple third feature maps are input into the feature fusion network for feature fusion and analysis to obtain the detection results.

[0150] It should be noted that each lightweight neural network module outputs a third feature map. Therefore, after feature extraction by multiple lightweight neural network modules, multiple third feature maps can be output. When a computer device obtains multiple third feature maps, it can input all of them into a feature fusion network for feature fusion and analysis to obtain the detection result. Optionally, the computer device can also input at least two target feature maps from the multiple third feature maps into the feature fusion network for feature fusion and analysis to obtain the detection result. The resolution of each target feature map is different. Different lightweight neural network modules can output feature maps with different resolutions. The stride of the channel-wise convolution in the lightweight neural network module is related to the resolution of the feature map output by the lightweight neural network module.

[0151] The above embodiments provide a specific backbone network, which includes multiple lightweight neural network modules. Compared with traditional backbone networks used for feature extraction, it greatly reduces the amount of computation and the required computing resources, thereby greatly shortening the computation time of the abnormal image detection model and improving the detection speed of the abnormal image detection model.

[0152] In one embodiment, a feature fusion network is provided, such as Figure 9 As shown, the feature fusion network includes a pointwise convolutional module 210, a first channelwise convolutional module 220, and a fusion layer 230. The pointwise convolutional module 210 is used to extract features from multiple feature maps output by the backbone network to obtain multiple fourth feature maps. The first channelwise convolutional module 220 is used to extract features from multiple fourth feature maps to obtain multiple fifth feature maps. The fusion layer 230 is used to fuse and analyze multiple fifth feature maps to obtain detection results.

[0153] Optionally, the above-mentioned pointwise convolution module 210 includes at least two fourth pointwise convolution layers 2100, the first channelwise convolution module 220 includes at least two third channelwise convolution layers 2200, the fourth pointwise convolution layer 2100 is connected to the corresponding third channelwise convolution layer 2200, and each third channelwise convolution layer 2200 is connected to the fusion module 230.

[0154] Optionally, the above-mentioned pointwise convolutional module includes N fourth pointwise convolutional layers, and the first channelwise convolutional module includes N third channelwise convolutional layers; the (N-1)th fourth pointwise convolutional layer and the (N-1)th third channelwise convolutional layer are connected by a first adder; N is greater than or equal to 2.

[0155] The first adder is used to fuse the (N-1)th pointwise feature map and the Nth pointwise feature map and output the result to the (N-1)th third-channel convolutional layer; the (N-1)th pointwise feature map is the feature map output by the (N-1)th pointwise convolutional layer, and the Nth pointwise feature map is the feature map output by the Nth fourth pointwise convolutional layer after upsampling.

[0156] Example illustration Figure 9 The feature fusion network shown is as follows: Figure 10 As shown, the pointwise convolutional module 210 in the feature fusion network includes three fourth pointwise convolutional layers (Pointwise Conv layers), and the first channel-wise convolutional module 220 includes three third channel-wise convolutional layers (Depthwise Conv layers), which are composed of a Pointwise Conv layer, an upsample layer, a Depthwise Conv layer, and a downsample layer in sequence. For the input feature maps C1, C2, and C3, they first pass through the Pointwise Conv layer to obtain feature maps P1, P2, and P3; P1+Upsample(P2), P2+Upsample(P3), and P3 are input to their respective Depthwise Conv layers to obtain feature maps D1, D2, and D3; D1, D2+downsample(D1), and D3+downsample(D2) are then input to obtain D`1, D`2, and D`3, and finally D`1, D`2, and D`3 are input to the Concat fusion layer for concatenation and analysis to obtain the final output result. It should be noted that this type of feature fusion network can also be called a feature pyramid.

[0157] In one embodiment, another feature fusion network is provided, such as Figure 11 As shown, the feature fusion network also includes a second channel-wise convolutional module 240, which may include at least two fourth channel-wise convolutional layers 2400.

[0158] Specifically, the second progressive convolution module 240 includes N fourth progressive convolution layers 2400; the Nth third progressive convolution layer 2300 and the Nth fourth progressive convolution layer 2400 are connected through a second adder.

[0159] The second adder is used to fuse the (N-1)th channel-wise feature map and the Nth channel-wise feature map and output them to the Nth fourth channel-wise convolutional layer; the (N-1)th channel-wise feature map is the feature map output by the (N-1)th third channel-wise convolutional layer after downsampling, and the Nth channel-wise feature map is the feature map output by the Nth third channel-wise convolutional layer.

[0160] Example illustration Figure 11 The feature fusion network shown is as follows: Figure 12 As shown, the fourth pointwise convolutional module 210 in the feature fusion network includes three fourth pointwise convolutional layers 2100 (Pointwise Conv layers), the first channel-wise convolutional module 220 includes three third channel-wise convolutional layers 2200 (Depthwise Conv layers), and the second channel-wise convolutional module 240 includes three fourth channel-wise convolutional layers 2400 (Depthwise Conv layers), which are composed of a Pointwise Conv layer, an Upsample layer, a Depthwise Conv layer, and a Downsample layer in sequence. For the input feature maps C1, C2, and C3, they are first processed through a Pointwise Conv layer to obtain feature maps P1, P2, and P3. P1+Upsample(P2), P2+Upsample(P3), and P3 are then input into their respective Depthwise Conv layers to obtain feature maps D1, D2, and D3. D1, D2+Downsample(D1), and D3+Downsample(D2) are then processed to obtain D`1, D`2, and D`3, which are input into subsequent Depthwise Conv layers to obtain outputs E1, E2, and E3. Finally, downsample(E1)+downsample(E2)+E3 is calculated to obtain the final output. In the output, each channel represents the confidence score of a class, used for loss calculation during training and confidence classification during inference deployment. Analyzing this confidence score further yields the detection results.

[0161] The above embodiments provide a feature fusion network, which includes multiple pointwise convolutional modules and channelwise convolutional modules for convolutional fusion processing. Furthermore, cross-feature fusion is performed between the pointwise convolutional modules and channelwise convolutional modules, which enhances the feature information and improves the accuracy of the final output detection result. This reduces the false detection rate of the abnormal image detection model containing this feature fusion network.

[0162] In practical applications, the aforementioned Figures 2-12 The abnormal image detection models used in the embodiments are all trained; therefore, this application also provides a method for training abnormal image detection models, such as... Figure 13 As shown, the method includes:

[0163] S301, Obtain the sample image set.

[0164] The sample image set includes various types of anomalous images, which can be images with abnormal shapes or colors. For example, the sample image set includes various types of distorted images such as stripes, tears, image residuals, mosaics, minor distorted images, and green screens.

[0165] In this embodiment, the computer device can download images to search for various types of abnormal images from a cloud database to obtain a sample image set. Optionally, the computer device can also pre-construct a sample image set by generating various types of abnormal images based on several normal images or videos.

[0166] S302, Train the initial neural network based on the sample image set to obtain the abnormal image detection model.

[0167] The initial neural network includes an initial backbone network, an initial feature fusion network connected to the initial backbone network, and a teacher network (see [link to original text]). Figure 14 ).

[0168] In this embodiment, the computer device pre-constructs an initial neural network, including constructing an initial backbone network, an initial feature fusion network, and a teacher network. Refer to the foregoing... Figures 2-12 The various network architectures described in the embodiments are constructed. The architectures of the teacher network and the initial feature fusion network are basically the same (see...). Figures 9-12 The teacher network assists in training the initial backbone network and the initial feature fusion network. Once the initial neural network is constructed, the sample image set can be input into it for training. The resulting network model is the anomaly image detection model used in the aforementioned embodiments. The training method provided in this embodiment, because the sample image set contains various types of anomaly images (i.e., the sample image set is representative), allows the anomaly image detection model trained based on this sample image set to detect multiple types of anomaly images, reducing the probability of false detections and making its application extremely wide-ranging.

[0169] Optionally, specific training methods are provided, such as Figure 15 As shown, the training method includes:

[0170] S401, input the sample images from the sample image set into the initial backbone network to obtain the sample feature map.

[0171] S402, input the sample feature map into the initial feature fusion network to obtain the first sample detection result.

[0172] S403 removes the gradient information from the sample feature map and inputs the removed sample feature map into the teacher network to obtain the second sample detection result.

[0173] S404. The initial backbone network and the initial feature fusion network are trained based on the detection results of the first sample and the detection results of the second sample to obtain the abnormal image detection model.

[0174] In the above method, the initial backbone network extracts features from the sample images and outputs multiple feature maps to the initial feature fusion network. The initial feature fusion network fuses and analyzes the multiple feature maps and outputs the first sample detection result. Correspondingly, after outputting multiple feature maps, the initial backbone network can further remove gradient information from each feature map, so that each feature map does not contain gradient information. This allows the teacher network to participate in training and affect the output loss, but does not affect the backpropagation of the model. The feature maps with gradient information removed are then passed through the teacher network to output the second sample detection result.

[0175] During the training process, after each input sample image, the loss function can be determined based on the detection results of the first sample and the second sample. The loss function can be found in the following relationship (1):

[0176]

[0177] Where p is the confidence level of the predicted sample image as an anomalous image, corresponding to the first sample detection result in this embodiment; q is the confidence level of the teacher network predicting the sample image as an anomalous image, corresponding to the second sample detection result in this embodiment; y represents the true label of the sample; α and γ are hyperparameters, with the default initial values ​​of α = 0.25 and γ = 2.

[0178] Training ends when the initial neural network reaches its expected performance or the maximum number of iterations is reached, resulting in a trained anomaly image detection model. The initial neural network reaching its expected performance can be achieved by the loss function converging or the loss function's output meeting preset conditions.

[0179] The training method described in the above embodiments, by employing a teacher network to assist the initial backbone network and initial feature fusion network in training, can accelerate the convergence speed of the loss, thereby improving the training speed. In summary, the anomaly image detection model provided in this application can be used to detect various types of anomaly images, with fast detection speed, low resource requirements, and extremely high detection accuracy. It is suitable for any type of device, especially for deployment in edge devices, thus demonstrating its wide applicability.

[0180] Next, the abnormal image detection model constructed based on the aforementioned embodiments was experimentally verified. Experiments showed that, trained on a training set of 40,000 images, it achieved excellent results on a test set of 10,000 images and 1,000 real video frames. The results are expressed in terms of precision, recall, and inference time at different input feature map resolutions. The specific calculation methods are shown in the following relationships (2) and (3):

[0181]

[0182]

[0183] Among them, TP is the case where the predicted video frame (normal image) is actually a normal video frame; FP is the case where the predicted video frame is actually a normal video frame (abnormal image); and FN is the case where the predicted video frame is actually an abnormal video frame.

[0184] Inference time is measured on the mobile device and is the time consumed from the moment the image is preprocessed and fed into the anomaly detection model to the moment the anomaly detection model outputs its detection result. Specific metrics are shown in Tables 1 and 2 below.

[0185] Table 1. Precision and Recall at Different Input Resolutions

[0186] Input resolution (FP32) Accuracy Recall rate 320x320 92.3% 96.3% 352x352 92.7% 96.5% 384x384 93.5% 97.2% 416x416 94.8% 97.3%

[0187] Table 2. Inference time for different input resolutions on different platforms

[0188]

[0189] In summary, this application designs a lightweight neural network module for use in anomaly image detection models. It can also be applied to general neural network structures, replacing computationally complex modules and providing a simple and efficient lightweight model solution. This significantly reduces training and inference computational resources without substantially decreasing performance, making it suitable for deployment on edge devices with limited computing resources. Furthermore, this application provides a solution and example for solving the screen distortion detection problem, addressing the issues of poor versatility and practicality in current screen distortion detection tasks. This solution effectively solves the problems of past solutions being unable to simultaneously detect multiple types of screen distortion and the occurrence of false positives and false negatives. It also provides a lightweight model that allows the solution to be trained and deployed to edge platforms with fewer computational resources. The practicality and feasibility of this solution are significantly improved compared to previous approaches.

[0190] In the actual training of the aforementioned anomaly image detection model, a large number of representative or highly generalizable sample images are often needed to expand the detection range and improve the detection accuracy. However, in practical applications, anomaly images, especially those with distorted or garbled images, are relatively rare, making it difficult to provide a large sample set for deep learning methods. Therefore, methods for generating anomaly images have emerged.

[0191] In methods for generating abnormal images, such as those for generating distorted or garbled images, the current main approach involves resolution compression and keyframe removal. Simply put, this method includes: acquiring video playback data; converting the video playback data to obtain compressed data; modifying the resolution of the compressed data and obtaining first video data; or discarding keyframes from the first compressed data and obtaining second video data; and generating sample distorted or garbled images based on the first or second video data. However, this method can only generate a limited number of distorted or garbled images, and the range of generated images is relatively large, lacking representativeness of the distorted or garbled image problem and failing to provide generalized distorted or garbled image samples for practical applications.

[0192] To address the aforementioned problems, this application further proposes a method for generating abnormal images, which can generate various types of distorted images. For example, the generated sample image set includes multiple striped distorted images, mosaic distorted images, torn distorted images, motion residual distorted images, artifact distorted images, and green screen distorted images. The following embodiments specifically describe this method.

[0193] In one embodiment, such as Figure 16 As shown, a method for generating abnormal images is provided, which uses... Figure 1 The computer device in the process is the executing entity, and the method includes the following steps:

[0194] S1101, Process the target data packet corresponding to the first image in the initial image set to obtain an image with an abnormal shape.

[0195] The initial image set can contain several normal images with identical content, or several normal images with different content. Optionally, the initial image set can be any discontinuous set of images, or it can be a set of several consecutive frames contained in a video. Some images in the initial image set can be regarded as the first image, which is actually the image to be processed by the data packet. The target data packet is the Nth data packet among several data packets corresponding to the first image. The Nth data packet can be a randomly selected data packet, and it is also a specific data packet, which is not limited in this application. The shape-abnormal image is an image containing at least one of the following shape abnormalities: stripes, mosaic, tearing, motion residuals, etc.

[0196] In this embodiment, the computer device can obtain an initial image set by downloading images or by directly obtaining the initial image set corresponding to the video. Once the computer device obtains the initial image set, it can process the target data packet of each first image in the image set to convert the first image into a shape-abnormal image. For example, it can first obtain several data packets corresponding to the first image, then determine the target data packet from these packets. It can randomly select a data packet as the target data packet, or it can specify a data packet as the target data packet. Then, it can perform any one or more processing actions on the target data packet, such as modification, discarding, or rearranging, so that the image generated subsequently based on the processed data packets contains shape-abnormal information, thus converting the first image into a shape-abnormal image.

[0197] It should be noted that when processing each first image in the initial image set, the computer device can use the same processing method to obtain the same type of shape anomalous image. For example, after processing ten first images using the same processing method, all ten will result in shape anomalous images containing mosaic. Alternatively, the computer device can use different processing methods to obtain different types of shape anomalous images. For example, after processing four first images using different processing methods, one will result in a shape anomalous image containing mosaic, one in a shape anomalous image containing stripes, one in a shape anomalous image containing tearing, and one in a shape anomalous image containing motion residuals. Alternatively, the computer device can also use different processing methods to process different groups of first images separately. For example, processing ten first images using the first processing method will result in ten shape anomalous images containing mosaic, and processing another ten first images using the second processing method will result in ten shape anomalous images containing tearing.

[0198] S1102, Process the pixel values ​​of the second image in the initial image set to obtain a color anomaly image.

[0199] In this context, some images in the initial image set can be considered as second images. Optionally, the second image can be any of the remaining images in the initial image set other than the first image, or it can be the same image as the first image, which is actually the image to be processed for pixel values. Color anomaly images, in practical applications, can manifest as images containing one or more colors of pixels, such as green, red, or blue.

[0200] In this embodiment, after the computer device acquires the initial image set, it can process the pixel values ​​of each second image in the image set to convert the second image into a color-abnormal image. Optionally, the computer device can also process the pixel values ​​of the second images after processing the first images in the initial image set to obtain the color-abnormal image. For example, the computer device can first identify one or more abnormal regions and set a reference pixel value as the abnormal pixel value. Then, it can use the set abnormal pixel value to replace all pixel values ​​within the abnormal region, thus modifying the pixel values ​​within the abnormal region. The image obtained after modification is the color-abnormal image.

[0201] It should be noted that when multiple abnormal regions exist on the second image, different abnormal pixel values ​​can be set for different abnormal regions. Furthermore, when processing each second image in the initial image set, the computer device can use the same processing method to obtain the same type of color abnormality image. For example, after processing ten second images using the same method, all ten will result in color abnormality images containing a green screen. Optionally, the computer device can also use different processing methods to obtain different types of color abnormality images. For example, after processing two second images using different methods, one will result in a color abnormality image containing a green screen, and the other in a color abnormality image containing a blue screen. Optionally, the computer device can also use different processing methods to process different groups of second images separately. For example, processing ten second images using the first processing method will result in ten color abnormality images containing a green screen, and processing another ten second images using the second processing method will result in ten blue screen abnormality images containing a blue screen.

[0202] S1103, combine the shape-abnormal images and color-abnormal images into a sample image set.

[0203] The sample image set is used to train the anomaly image detection model. The anomaly image detection model is used to detect whether an image is an anomaly, or to detect the type of anomalous information contained in an image.

[0204] In this embodiment, when the computer device obtains multiple abnormal shape images and multiple abnormal color images based on the aforementioned steps, the image set composed of these multiple abnormal shape images and multiple abnormal color images can be used as a sample image set. Then, the computer device can train a neural network model or other network model based on the sample image set to obtain an abnormal image detection model.

[0205] The above-described method for generating abnormal images involves processing the target data packet corresponding to the first image in the initial image set to obtain a shape-abnormal image, and processing the pixel values ​​of the second image in the initial image set to obtain a color-abnormal image. The shape-abnormal image and the color-abnormal image are then combined to form a sample image set, which can be used to train an abnormal image detection model. This method can generate various types of abnormal images, meaning the sample image set contains a variety of representative abnormal images, thereby improving the generalization ability of the sample image set and providing reliable data support for subsequent training of the abnormal image detection model.

[0206] In one embodiment, a specific implementation method for processing the target data packet is provided, namely, the above-mentioned S1101 "processing the target data packet corresponding to the first image in the initial image set to obtain an image with abnormal shape", such as Figure 17 As shown, it includes:

[0207] S2201, determine the target data packet from all data packets of the first image according to the preset data packet interval.

[0208] The data packet interval can be determined according to actual processing needs. For example, the data packet interval can be 100 data packets per interval or 1000 data packets per interval.

[0209] In this embodiment, when the computer device acquires the first image, it can convert the first image into several corresponding data packets, and determine the target data packet among these data packets according to a preset data packet interval, for example, determining one target data packet every 100 data packets. Optionally, the computer can also use a connectionless transmission protocol (UDP) to push and pull video locally, and determine the target data packet in the data packets corresponding to the video image during the push and pull streaming process. Specifically, the computer device can set up ports A and B locally, push the normal video image from port A to port B in the form of a UDP stream, receive the data packets corresponding to the video image from port B, and determine the target data packet at port B according to the preset data packet interval received, for example, the 101st data packet is taken as the target data packet every 100 data packets received, or the 1001st data packet is taken as the target data packet every 1000 data packets received. Optionally, the computer device can also set up port C locally. After processing the target data packet on port B, the processed UDP stream can be pushed to port C, which receives and saves the video locally, resulting in a video containing the abnormal image.

[0210] S2202, Determine the target byte from the target data packet.

[0211] The target byte can be any byte in the target data packet or a specified byte within the target data packet. In this embodiment, once the computer device determines the target data packet, it can randomly select a byte from the several bytes contained in the target data packet, or specify a byte as the target byte, for subsequent processing. It is understood that the specified byte can be determined according to actual processing requirements; for example, the specified byte could be the 100th byte or the 200th byte in the target data packet. It should be noted that when randomly selecting bytes from the several bytes corresponding to each target data packet, the target bytes in the multiple target data packets are randomly distributed, and this random distribution can be uniform.

[0212] S2203, perform anomaly processing on the target byte to obtain an image with an abnormal shape.

[0213] Exception handling can include any one or a combination of processes such as modifying, discarding, or rearranging bytes.

[0214] In this embodiment, the computer device can perform one or more combined anomaly processing operations on the target bytes, such as modification, discarding, or rearranging, to simulate data errors during data transmission and thus obtain an image with an abnormal shape. It should be noted that when the computer device performs anomaly processing on a target byte at preset data packet intervals, the same anomaly processing method can be used, for example, modifying the data packet for all bytes. Alternatively, different anomaly processing methods can be used; for example, the first few target bytes can be processed by modifying the data packet, while the later target bytes can be processed by discarding the data packet.

[0215] Optionally, when performing anomaly processing on the target byte, the computer device can specifically invert at least one bit in the target byte to obtain an image with an abnormal shape. For example, for all received data packets, every 120 data packets, a byte is randomly selected as the target byte, and then 1 to 2 bits from bits 0-7 of the target byte are randomly selected and inverted to simulate an error in the data transmission process.

[0216] Optionally, when performing anomaly processing on the target byte, the computer device can also directly discard the target byte to obtain an image with an abnormal shape. For example, for all received data packets, every 1000 data packets, a byte can be randomly selected as the target byte and then discarded to simulate packet loss during data transmission.

[0217] Optionally, when performing exception handling on the target byte, the computer device can select different exception handling methods based on the type of the target byte. The type of the target byte is determined by the data packet interval at which it is identified. For example, if a target byte is identified every 120 data packets, it is considered an error byte; if a target byte is identified every 1000 data packets, it is considered a lost byte. Different exception handling methods can be selected for different types of target bytes. For instance, for all received data packets, every 120 data packets, a byte is randomly selected as the error byte target byte, and 1 to 2 bits from bits 0-7 of that target byte are randomly selected and inverted to simulate a data transmission error. Similarly, every 1000 data packets, a byte is randomly selected as the lost byte target byte and discarded to simulate packet loss during data transmission.

[0218] The above embodiments achieve image anomalies by performing anomaly processing on the bytes of the data packets corresponding to the image, thereby simulating errors or loss during data packet transmission and obtaining shape-abnormal images. This method is simple and practical, and can obtain various types of shape-abnormal images, providing representative data support for enriching the sample image set in the later stage.

[0219] In one embodiment, a specific implementation method for processing pixel values ​​is provided, namely, the above-mentioned S1102 "processing the pixel values ​​of the second image in the initial image set to obtain a color anomaly image", such as Figure 18 As shown, it includes:

[0220] S3301, Determine the target pixel value of the second image based on the color channel type of the second image.

[0221] Among them, color channel type refers to the color encoding method of the second image. For example, some images can use the optical three primary colors (Red, Green, Blue, RGB) color encoding method to represent the color of each pixel in the image, while some images can use the hexagonal pyramid model (Hue Saturation Vlaue) color encoding method to represent the color of each pixel in the image.

[0222] In this embodiment, when the computer device acquires the second image, it can further determine the color channel type of the second image, such as whether the color channel type of the second image is RGB or HSV. Then, based on the different color channel types, it can determine the target pixel value of each pixel on the corresponding second image, so that the color of the target pixel value on the second image is closer to the abnormal color in the actual image, such as the green screen color.

[0223] S3302, set the pixel value of the target connected component in the second image to the target pixel value to obtain the color anomaly image.

[0224] The target connected component is the region with color anomalies, which can be preset by the computer device. For example, the size of the connected component can be set to {w, h}, where w ∈ [W min W max ], h∈[H min H max ]. [W min W max ] represents the upper and lower limits of the length of the target connected component, and [H] represents the upper and lower limits of the length of the target connected component. min H max [] represents the upper and lower limits of the width of the target connected component.

[0225] In this embodiment, after the computer device sets the target pixel value, it can further determine the target connected region, and then replace the original pixel values ​​in the target connected region with the target pixel value, causing the image color contained in the target connected region to appear abnormal, thus obtaining a color-abnormal image. This method is simple and practical, requiring only the modification of the pixel values ​​of pixels in the target connected region to achieve color anomaly processing, thereby obtaining a color-abnormal image, providing representative data support for enriching the sample image set later.

[0226] In practical applications, the color channel types of general images include two types: HSB and RGB. Here, we provide methods for determining the target pixel value for each of these two types.

[0227] The first method, for the HSB type, specifically includes S3301 "determining the target pixel value of the second image according to the color channel type of the second image", which includes: if the color channel type of the second image is HSV channel type, then the target pixel value of the H channel of the second image is set to a random value that satisfies the first normal distribution, the target pixel value of the S channel of the second image is set to a random value that satisfies the second normal distribution, and the pixel value of the V channel of the second image is corrected according to the preset weight to obtain the target pixel value of the V channel.

[0228] This method is illustrated by example. When the color channel type of the second image is HSV, the target pixel value of the H channel can be set to a random value h, which satisfies the first normal distribution h ~ N(60,5); the target pixel value of the S channel can be set to a random value s, which satisfies the second normal distribution s ~ N(250,5); for the V channel, a preset weight can be set first, and then the pixel value of the V channel can be modified using this preset weight to obtain the target pixel value v of the V channel. The modified target pixel value of the HSV channel is then (h, s, v). It should be noted that the specific selection of the first and second normal distributions can be determined based on actual experimental data. This is merely an example of an optimal method and does not constitute a limitation on these parameters. The above data can make the final color anomaly image closer to the actual color anomaly image.

[0229] Optionally, a method for determining the aforementioned preset weights is provided, such as... Figure 19 As shown, the method includes:

[0230] S4401, randomly determine the target parameter value from the preset parameter set.

[0231] The preset parameter set can be determined in advance by the computer device based on experimental data. For example, in this embodiment, the preset parameter set can be {0.7, 1.1, 1.9}, which is an optimal parameter set. The target parameter value is used to determine the preset weight. In this embodiment, the computer device can obtain a preset parameter set in advance and randomly select a value from the set as the target parameter value. It should be noted that the selection probabilities of each value in the preset parameter set can be the same or different. For example, if the preset parameter set is {0.7, 1.1, 1.9}, the corresponding selection probabilities are {0.4, 0.2, 0.4}.

[0232] S4402, randomly determine the preset weights under the third normal distribution.

[0233] The expected value of the third normal distribution is the target parameter value.

[0234] In this embodiment, when the computer device determines the target parameter value based on the aforementioned steps, it can use this target parameter value as the expected value to determine the third normal distribution. For example, the third normal distribution can be N(a, 0.2), where a is the target parameter value. Then, a parameter is randomly generated that satisfies the third normal distribution; this parameter is the preset weight. Using the preset weight determined by the above method to modify the pixel values ​​of the V channel in the second image, pixel values ​​that are more consistent with those in color anomaly images in actual applications can be obtained, thereby improving the realism of the generated color anomaly image.

[0235] The second type, for RGB type, the above S3301 "determine the target pixel value of the second image according to the color channel type of the second image" specifically includes: if the color channel type of the second image is RGB channel type, then the target pixel value of the R channel of the second image is set to a random value that satisfies the fourth normal distribution, the target pixel value of the B channel of the second image is set to a random value that satisfies the fifth normal distribution, and the original pixel value of the G channel of the second image is determined as the target pixel value of the G channel.

[0236] To illustrate this method, when the color channel type of the second image is RGB, the target pixel value of the R channel can be specifically set to a random value. The expected value is 0 and the variance is The fourth normal distribution, namely: Set the target pixel value of channel B to a random value. The expected value is 0 and the variance is The fifth normal distribution, namely The pixel value of the G channel remains unchanged, retaining its original pixel value g; therefore, the target pixel value of the RGB channel modified using this method is... It should be noted that the specific selection of the fourth and fifth normal distributions mentioned above can be determined based on actual experimental data. This is merely an example illustrating an optimal approach and does not constitute a limitation on these parameters. The above data can make the final color anomaly image closer to the actual color anomaly image.

[0237] In one embodiment, a specific implementation method is provided for modifying the pixel value of a target connected component using the target pixel value, namely, the above-mentioned S3302 "setting the pixel value of the target connected component in the second image to the target pixel value to obtain a color anomaly image", such as Figure 20 As shown, it includes:

[0238] S5501, Determine the identity matrix block centered at the initial coordinate point.

[0239] The identity matrix block can be predetermined by computer equipment, and its size can be specifically {W}. b H b}, W b H b All satisfy a uniform distribution, that is: W b ~U(50,100), H b ~U(50,100). The size of the identity matrix block affects the shape of the target connected region. For example, the smaller the identity matrix block, the smaller the corresponding granularity, and the corresponding target connected region may be jagged. The larger the identity matrix, the larger the corresponding granularity, and the corresponding target connected region may contain more right angles and edges.

[0240] In this embodiment, after the computer device determines the size of the target connected region, as described in S3302 above, for example, the computer device pre-sets the size of the target connected region to be {w,h}, where w∈[W min W max ], h∈[H min H max ], [W min W max ] represents the upper and lower limits of the length of the target connected component, and [H] represents the upper and lower limits of the length of the target connected component. min H max [ ] represents the upper and lower limits of the width of the target connected component. Then, an initial starting point (x, y) is selected, where the x-coordinate of this initial starting point satisfies a sixth normal distribution, which is related to the length of the second image. For example, the sixth normal distribution can be specifically... W is the length of the second image; the ordinate y of the initial starting point satisfies a seventh normal distribution, which is related to the width of the identity matrix block. For example, the seventh normal distribution can be specifically... H represents the width of the second image. Once an initial coordinate point is selected, a unit matrix block centered on that initial coordinate point can be further defined so that the pixel values ​​of the target connected region can be modified later based on this unit matrix block.

[0241] S5502 sets the pixel value of all pixels within the identity matrix block to the target pixel value.

[0242] Once the identity matrix block is determined, the pixel values ​​of all pixels within the identity matrix block can be modified using the previously determined target pixel values; a direct replacement operation can be performed.

[0243] S5503, move the unit matrix block according to a preset direction and a preset step size, and after each move, perform the step of setting the pixel value of all pixels in the unit matrix block to the target pixel value, until the area of ​​the connected region formed by all target pixel values ​​reaches the area of ​​the target connected region, and obtain the color anomaly image.

[0244] The preset step size includes a horizontal coordinate step size and a vertical coordinate step size. The horizontal coordinate step size is related to the width of the identity matrix block, and the vertical coordinate step size is related to the length of the identity matrix block. The preset direction can be specifically defined as movement along the positive direction of the horizontal coordinate, along the negative direction of the horizontal coordinate, along the positive direction of the vertical coordinate, and along the negative direction of the vertical coordinate in the Cartesian coordinate system of the image. For example, the preset step size for movement is (x... d ,y d ), (x d ,y d The following conditions must be met:

[0245]

[0246] In this embodiment, after the computer device modifies the pixel values ​​of all pixels within the current identity matrix block, it can further move the identity matrix block according to a preset direction and a preset step size, and continue to modify the pixel values ​​of all pixels within the moved identity matrix block until the area of ​​the connected regions corresponding to all modified pixel values ​​reaches the area of ​​the target connected region. At this point, the movement of the identity matrix block stops, and the image after modifying the pixel values ​​becomes a color-abnormal image. For example, if the size of the target connected region is set to {w, h}, and its area is S = w * h, then when the area of ​​the connected regions corresponding to all modified pixels traversed by the identity matrix block equals S, the movement of the identity matrix block stops, completing the modification of the pixel values ​​within the target connected region, making the image color presented by the pixel values ​​within the target connected region an abnormal color. It should be noted that while moving the identity matrix block, the initial coordinates within the identity matrix block are also being moved. After each movement, the specific coordinates for the next movement are calculated based on the preset direction and the current initial coordinates and placed in a queue. This allows for the next movement to retrieve usable coordinates from the queue and assign them to the initial coordinates within the current identity matrix block, thus realizing the movement of the identity matrix block.

[0247] Based on all the above embodiments, a method for generating abnormal images is provided, such as... Figure 21 As shown, the method includes:

[0248] S6601, for all data packets corresponding to the first image in the initial image set, determine the first target data packet according to a preset first data packet interval, and determine the second target data packet according to a preset second data packet interval, wherein the second data packet interval is greater than the first data packet interval.

[0249] S6602, determine the first target byte from the first target data packet, and determine the second target byte from the second target data packet.

[0250] S6603, invert at least one bit in the first target byte, discard the second target byte, and obtain an image with an abnormal shape.

[0251] S6604, determine the target pixel value of the second image based on the color channel type of the second image.

[0252] S6605, determine the size of the target connected region and the identity matrix block centered at the initial coordinate point on the second image.

[0253] S6606 sets the pixel value of all pixels within the identity matrix block to the target pixel value.

[0254] S6607, move the unit matrix block according to a preset direction and a preset step size, and execute the above steps S6606 after each move until the area of ​​the connected region formed by all target pixel values ​​reaches the area of ​​the target connected region, thus obtaining a color anomaly image.

[0255] S6608, which combines images with abnormal shapes and images with abnormal colors into a sample image set; the sample image set is used to train an abnormal image detection model.

[0256] Each of the above steps has been described in the foregoing; please refer to the aforementioned embodiments for details, which will not be repeated here. Furthermore, the above method can be applied to any video image to generate various abnormal images, including stripes, tearing, image residuals, mosaic, minor screen tearing, green screen, etc. For example, see... Figure 22 The images show various types of abnormalities.

[0257] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0258] Based on the same inventive concept, this application also provides an apparatus for detecting and generating abnormal images as described above. The solution provided by this apparatus is similar to the solution described in the above method; therefore, the specific limitations of one or more embodiments of the abnormal image detection and generation apparatus provided below can be found in the limitations of the abnormal image detection and generation methods described above, and will not be repeated here.

[0259] In one embodiment, such as Figure 23 As shown, an abnormal image detection device is provided, comprising:

[0260] The acquisition module 1000 is used to acquire the image to be detected;

[0261] The detection module 2000 is used to input the image to be detected into a preset abnormal image detection model for abnormal detection and obtain the detection result. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image.

[0262] Each module in the aforementioned abnormal image detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0263] In one embodiment, such as Figure 24 As shown, an apparatus for generating abnormal images is provided, comprising:

[0264] The first processing module 3000 is used to process the target data packet corresponding to the first image in the initial image set to obtain an image with an abnormal shape;

[0265] The second processing module 4000 is used to process the pixel values ​​of the second image in the initial image set to obtain a color anomaly image;

[0266] The construction module 5000 is used to form a sample image set by combining the shape abnormal image and the color abnormal image; the sample image set is used to train an abnormal image detection model.

[0267] Each module in the aforementioned abnormal image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0268] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0269] Acquire the image to be detected;

[0270] The image to be detected is input into a preset abnormal image detection model for anomaly detection, and the detection result is obtained. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image.

[0271] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0272] Acquire the image to be detected;

[0273] The image to be detected is input into a preset abnormal image detection model for anomaly detection, and the detection result is obtained. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image.

[0274] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0275] Acquire the image to be detected;

[0276] The image to be detected is input into a preset abnormal image detection model for anomaly detection, and the detection result is obtained. The detection model includes a backbone network and a feature fusion network for feature fusion and analysis. The backbone network uses pointwise convolution and channelwise convolution to extract features from the image.

[0277] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0278] The target data packet corresponding to the first image in the initial image set is processed to obtain an image with an abnormal shape;

[0279] The pixel values ​​of the second image in the initial image set are processed to obtain a color anomaly image;

[0280] The abnormal shape image and the abnormal color image are combined into a sample image set; the sample image set is used to train an abnormal image detection model.

[0281] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0282] The target data packet corresponding to the first image in the initial image set is processed to obtain an image with an abnormal shape;

[0283] The pixel values ​​of the second image in the initial image set are processed to obtain a color anomaly image;

[0284] The abnormal shape image and the abnormal color image are combined into a sample image set; the sample image set is used to train an abnormal image detection model.

[0285] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0286] The target data packet corresponding to the first image in the initial image set is processed to obtain an image with an abnormal shape;

[0287] The pixel values ​​of the second image in the initial image set are processed to obtain a color anomaly image;

[0288] The abnormal shape image and the abnormal color image are combined into a sample image set; the sample image set is used to train an abnormal image detection model.

[0289] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0290] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0291] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting abnormal images, characterized in that, The method includes: Acquire the image to be detected; The image to be detected is input into a preset abnormal image detection model for anomaly detection to obtain the detection result; the detection model includes a backbone network and a feature fusion network for feature fusion and analysis, and the backbone network uses pointwise convolution and channelwise convolution to extract features from the image; The backbone network includes multiple cascaded lightweight neural network modules. At least one of the lightweight neural network modules uses pointwise convolution and channelwise convolution to perform feature extraction on the image. The lightweight neural network module includes: a first branch, a second branch, and a channel splicing layer. The first branch is used to perform pointwise convolution and channelwise convolution on the image to obtain the first branch feature map; The second branch is used to perform global average pooling and channel-wise convolution on the image to obtain the second branch feature map; The channel splicing layer is used to splice the first branch feature map and the second branch feature map to obtain a spliced ​​feature map.

2. The method according to claim 1, characterized in that, The lightweight neural network module also includes a channel shuffling layer; The channel shuffling layer is used to shuffle the spliced ​​feature map.

3. The method according to claim 1, characterized in that, The first branch includes: a first pointwise convolutional layer, a first channelwise convolutional layer, and a second pointwise convolutional layer connected in sequence.

4. The method according to claim 1, characterized in that, The second branch includes a global pooling layer and a third pointwise convolutional layer connected in sequence.

5. The method according to claim 1, characterized in that, Each of the lightweight neural network modules further includes: a third branch and a branch adder; The third branch is used to perform channel-wise convolution on the image to obtain the third branch feature map; Correspondingly, the second branch is also used to perform size processing on the second branch feature map to obtain a processed feature map; the size of the processed feature map is consistent with the size of the third branch feature map; The branch adder is used to fuse the processed feature map and the third branch feature map to obtain the fourth branch feature map; Correspondingly, the channel splicing layer is used to splice the first branch feature map and the fourth branch feature map to obtain the spliced ​​feature map.

6. The method according to claim 5, characterized in that, The third branch includes a second channel-wise convolution.

7. The method according to any one of claims 1-6, characterized in that, The backbone network also includes multiple convolutional kernels and pooling layers. The step of inputting the image to be detected into a preset anomaly detection model for anomaly detection, and obtaining the detection result, includes: The image to be detected is extracted using the multiple convolutional kernels to obtain a first feature map; The first feature map is pooled using the pooling layer to obtain the second feature map; The multiple lightweight neural network modules are used to extract features from the second feature map to obtain multiple third feature maps; The multiple third feature maps are input into the feature fusion network for feature fusion and analysis to obtain the detection result.

8. The method according to claim 7, characterized in that, The step of inputting the third feature map into the feature fusion network for feature fusion and analysis to obtain the detection result includes: At least two target feature maps from the plurality of third feature maps are input into the feature fusion network for feature fusion and analysis to obtain the detection result; the resolution of each target feature map is different.

9. The method according to claim 1, characterized in that, The feature fusion network includes a pointwise convolutional module, a first channelwise convolutional module, and a fusion module; The pointwise convolution module is used to extract features from multiple feature maps output by the backbone network to obtain multiple fourth feature maps. The first channel-wise convolution module is used to extract features from the plurality of fourth feature maps to obtain a plurality of fifth feature maps; The fusion module is used to fuse and analyze the multiple fifth feature maps to obtain the detection result.

10. The method according to claim 9, characterized in that, The pointwise convolution module includes at least two fourth pointwise convolutional layers, and the first channelwise convolution module includes at least two third channelwise convolutional layers. The fourth pointwise convolutional layers are connected to the corresponding third channelwise convolutional layers, and each third channelwise convolutional layer is connected to the fusion module.

11. The method according to claim 10, characterized in that, The pointwise convolution module includes N fourth pointwise convolutional layers, and the first channelwise convolution module includes N third channelwise convolutional layers; the (N-1)th fourth pointwise convolutional layer and the (N-1)th third channelwise convolutional layer are connected by a first adder; N is greater than or equal to 2; The first adder is used to fuse the (N-1)th pointwise feature map and the Nth pointwise feature map and output the result to the (N-1)th third channelwise convolutional layer; the (N-1)th pointwise feature map is the feature map output by the (N-1)th fourth pointwise convolutional layer, and the Nth pointwise feature map is the feature map output by the Nth pointwise convolutional layer after upsampling.

12. The method according to claim 11, characterized in that, The feature fusion network further includes a second channel-wise convolutional module, which includes N fourth channel-wise convolutional layers; the Nth third channel-wise convolutional layer and the Nth fourth channel-wise convolutional layer are connected by a second adder. The second adder is used to fuse the (N-1)th channel-wise feature map and the Nth channel-wise feature map and output them to the Nth fourth channel-wise convolutional layer; the (N-1)th channel-wise feature map is the feature map output by the (N-1)th third channel-wise convolutional layer after downsampling, and the Nth channel-wise feature map is the feature map output by the Nth third channel-wise convolutional layer.

13. The method according to claim 11, characterized in that, The method for training the abnormal image detection model includes: Obtain a set of sample images; The initial neural network is trained based on the sample image set to obtain the abnormal image detection model; the initial neural network includes an initial backbone network, an initial feature fusion network and a teacher network respectively connected to the initial backbone network.

14. The method according to claim 13, characterized in that, The initial neural network is trained based on the sample image training set to obtain the abnormal image detection model; The sample images in the sample image set are input into the initial backbone network to obtain sample feature maps; The sample feature map is input into the initial feature fusion network to obtain the first sample detection result; The gradient information of the sample feature map is removed, and the removed sample feature map is input into the teacher network to obtain the second sample detection result; The initial backbone network and the initial feature fusion network are trained based on the detection results of the first sample and the detection results of the second sample to obtain the abnormal image detection model.

15. A method for generating an abnormal image, characterized in that, The method includes: The target data packet corresponding to the first image in the initial image set is processed to obtain an image with an abnormal shape; The pixel values ​​of the second image in the initial image set are processed to obtain a color anomaly image; The abnormal shape image and the abnormal color image are combined into a sample image set; the sample image set is used to train the abnormal image detection model as described in any one of claims 1-14.

16. The method according to claim 15, characterized in that, The process of processing the target data packet corresponding to the first image in the initial image set to obtain the shape-abnormal image includes: The target data packet is determined from all data packets of the first image according to a preset data packet interval; Determine the target byte from the target data packet; The target byte is subjected to anomaly processing to obtain the shape-abnormal image.

17. The method according to claim 16, characterized in that, The step of performing anomaly processing on the target bytes to obtain the shape-abnormal image includes: Invert at least one bit in the target byte to obtain the abnormal shape image.

18. The method according to claim 16, characterized in that, The step of performing anomaly processing on the target bytes to obtain the shape-abnormal image includes: Discard the target byte to obtain the image with the abnormal shape.

19. The method according to any one of claims 15-18, characterized in that, The step of processing the pixel values ​​of the second image in the initial image set to obtain a color anomaly image includes: The target pixel value of the second image is determined based on the color channel type of the second image; The pixel value of the target connected component in the second image is set as the target pixel value to obtain the color anomaly image.

20. The method according to claim 19, characterized in that, Determining the target pixel value of the second image based on its color channel type includes: If the color channel type of the second image is HSV channel type, then the target pixel value of the H channel of the second image is set to a random value that satisfies the first normal distribution, the target pixel value of the S channel of the second image is set to a random value that satisfies the second normal distribution, and the pixel value of the V channel of the second image is corrected according to a preset weight to obtain the target pixel value of the V channel.

21. The method according to claim 20, characterized in that, The method for determining the preset weights includes: Randomly determine the target parameter value from the preset parameter set; The preset weights are randomly determined under a third normal distribution; the expected value of the third normal distribution is the target parameter value.

22. The method according to claim 19, characterized in that, Determining the target pixel value of the second image based on its color channel type includes: If the color channel type of the second image is RGB, then the target pixel value of the R channel of the second image is set to a random value that satisfies the fourth normal distribution, the target pixel value of the B channel of the second image is set to a random value that satisfies the fifth normal distribution, and the original pixel value of the G channel of the second image is determined as the target pixel value of the G channel.

23. The method according to claim 19, characterized in that, The step of setting the pixel value of the target connected component in the second image to the target pixel value to obtain the color anomaly image includes: Determine the identity matrix block centered at the initial coordinate point; Set the pixel value of all pixels within the identity matrix block to the target pixel value; The identity matrix block is moved in a preset direction and with a preset step size. After each movement, the step of setting the pixel value of all pixels in the identity matrix block to the target pixel value is performed until the area of ​​the connected region formed by all target pixel values ​​reaches the area of ​​the target connected region, thereby obtaining the color anomaly image.

24. The method according to claim 23, characterized in that, The x-coordinates of the initial coordinate points follow a sixth normal distribution, which is related to the length of the second image. The ordinate of the initial coordinate point satisfies the seventh normal distribution; the seventh normal distribution is related to the width of the second image.

25. The method according to claim 23, characterized in that, The preset step size includes a horizontal coordinate step size and a vertical coordinate step size. The horizontal coordinate step size is related to the width of the identity matrix block, and the vertical coordinate step size is related to the length of the identity matrix block.

26. An anomalous image detection device, characterized in that, The detection device includes: The acquisition module is used to acquire the image to be detected; The detection module is used to input the image to be detected into a preset abnormal image detection model for abnormal detection and obtain the detection result; the detection model includes a backbone network and a feature fusion network for feature fusion and analysis, and the backbone network uses pointwise convolution and channelwise convolution to extract features from the image; The backbone network includes multiple cascaded lightweight neural network modules. At least one of the lightweight neural network modules uses pointwise convolution and channelwise convolution to perform feature extraction on the image. The lightweight neural network module includes: a first branch, a second branch, and a channel splicing layer. The first branch is used to perform pointwise convolution and channelwise convolution on the image to obtain the first branch feature map; The second branch is used to perform global average pooling and channel-wise convolution on the image to obtain the second branch feature map; The channel splicing layer is used to splice the first branch feature map and the second branch feature map to obtain a spliced ​​feature map.

27. An apparatus for generating an abnormal image, characterized in that, The device includes: The first processing module is used to process the target data packet corresponding to the first image in the initial image set to obtain an image with an abnormal shape; The second processing module is used to process the pixel values ​​of the second image in the initial image set to obtain a color anomaly image; A construction module is used to form a sample image set by combining the shape abnormal image and the color abnormal image; the sample image set is used to train an abnormal image detection model as described in any one of claims 1-14.

28. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 25.

29. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 25.

30. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 25.

Citation Information

Patent Citations

  • Multilevel feature pyramid target detection method based on depth separable convolution and fusion PAN

    CN113111736A

  • compact 3D convolution-based lip language recognition method, system, device and storage medium

    CN113313056A