Target detection method and related device

Abstract

The invention provides a target detection method and a related device. The target detection method is a method for performing target detection on a to-be-detected image in a video by using a target detection network, the target detection network comprises a feature extraction module, an inter-frame feature reconstruction module and a detection module, and the method comprises the steps of performing feature extraction on the to-be-detected image by using the feature extraction module to obtain a first feature; reconstructing the first feature by using the inter-frame feature reconstruction module to obtain a second feature; and performing detection processing on the second feature by using the detection module to obtain a target detection result of the to-be-detected image. According to the invention, the problem of low detection result stability of an image target detection algorithm in a video target detection scene can be solved.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and in particular to target detection methods and related apparatus. Background Technology

[0002] When image target detection algorithms are directly applied to video target detection, the presence of moving images or rapidly moving targets in the video scene can lead to temporary blurring or bokeh of the target or background, resulting in a degradation in image quality. Furthermore, image target detection algorithms only consider the features of the current image, and their detection performance is directly affected by the image quality. Therefore, the above scenarios will directly affect the accuracy and detection rate of target detection. For example, if the same target is located in the same scene and most frames can be detected normally within the same time period, some frames may be missed. Summary of the Invention

[0003] This application provides a target detection method and related apparatus to solve the problem of low stability of detection results in image target detection algorithms in video target detection scenarios.

[0004] To address the aforementioned objectives, this application provides a target detection method. This method utilizes a target detection network to detect targets in a video image. The target detection network includes a feature extraction module, an inter-frame feature reconstruction module, and a detection module. The method comprises: extracting features from the image to be detected using the feature extraction module to obtain a first feature; reconstructing the first feature using the inter-frame feature reconstruction module to obtain a second feature; and performing detection processing on the second feature using the detection module to obtain a target detection result for the image to be detected.

[0005] To address the aforementioned problems, this application provides an electronic device, which includes a processor; the processor is used to execute instructions to implement the steps of the above-described method.

[0006] To address the aforementioned problems, this application provides a computer storage medium storing instruction / program data, which, when executed, implements the steps of the method described above.

[0007] The method of this application is as follows: After the target detection method extracts the first feature from the image to be detected in the video using the feature extraction module, it can reconstruct the first feature using the inter-frame feature reconstruction module to obtain the second feature. Then, the target detection result of the image to be detected is obtained by using the detection module to detect the second feature. This allows the high similarity between video frames to reconstruct or enhance the feature representation of the current frame, so that the inter-frame feature reconstruction module can reconstruct the information in the current frame that is useful for the next frame, thereby achieving feature denoising and improving the detection accuracy of the detection module. This solves the problem of low detection result stability in video target detection scenarios. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings that can be used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating the implementation method of the target detection method of this application; Figure 2 This is a schematic diagram of the inter-frame feature reconstruction module in the target detection method of this application; Figure 3 This is a schematic diagram of the encoder part of the inter-frame feature reconstruction module in the target detection method of this application; Figure 4 This is a schematic diagram of the target detection network in the target detection method of this application; Figure 5 This is a schematic diagram of the decoder part of the inter-frame feature reconstruction module in the target detection method of this application; Figure 6 This is a schematic diagram of the structure of the electronic device of this application; Figure 7 This is a schematic diagram of one embodiment of the computer storage medium of this application. Detailed Implementation

[0010] To enable those skilled in the art to better understand the technical solution of this application, the target detection method and related apparatus provided in this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0011] The terms "first," "second," and "third" used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0012] In this document, the term "implementation" means that a specific feature, structure, or characteristic described in connection with an implementation may be included in at least one implementation of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same implementation, nor is it a separate or alternative implementation mutually exclusive with other implementations. It will be explicitly and implicitly understood by those skilled in the art that, without conflict, the implementations described herein may be combined with other implementations.

[0013] The object detection method of this application is a method for detecting objects in a video image using an object detection network, wherein the object detection network includes a feature extraction network, an inter-frame feature reconstruction module, and a detection module. Figure 1 As shown, Figure 1 This is a flowchart illustrating one embodiment of the target detection method of this application. The target detection method of this application may include the following steps.

[0014] S11: Use the feature extraction module to extract features from the image to be detected to obtain the first feature.

[0015] Optionally, the feature extraction module can be used to extract features from the image to be detected in the video to obtain the first feature, so that the target detection can be performed based on the first feature in the subsequent process.

[0016] The structure of the feature extraction module is not restricted, as long as it has the function of feature extraction.

[0017] In one implementation, the feature extraction module includes a backbone network and a neck structure. The backbone network extracts image features at multiple scales from the image to be detected, and the neck structure further refines the multi-scale image features to obtain the deep semantic features of the image, which is the first feature of the image to be detected.

[0018] In another implementation, the feature extraction module comprises a single backbone network. For example, a pre-trained deep convolutional neural network (such as ResNet, VGG, etc.) can be used to extract features directly from the input image.

[0019] In another implementation, the feature extraction module may include a Feature Pyramid Network (FPN). For example, FPN generates multi-level feature representations by fusing feature maps of different scales, which helps in handling targets of different sizes. It is widely used in object detection tasks, especially performing well in multi-scale object detection.

[0020] Optionally, in step S11, the image to be detected can be input to the feature extraction module so that the feature extraction module can extract features from the image to be detected and obtain the first feature.

[0021] S12: The first feature is reconstructed using the inter-frame feature reconstruction module to obtain the second feature.

[0022] After extracting the first feature from the image to be detected in the video using the feature extraction module, the first feature can be reconstructed using the inter-frame feature reconstruction module to obtain the second feature. This second feature can then be used by the detection module to detect the target in the image to be detected. This allows the high similarity between video frames to reconstruct or enhance the feature representation of the current frame, enabling the inter-frame feature reconstruction module to reconstruct information in the current frame that is useful for the next frame. This achieves feature denoising and improves the detection accuracy of the detection module, thereby solving the problem of low detection result stability in video target detection scenarios.

[0023] In one implementation, spatiotemporal information between consecutive frames in the video can be used to extract richer feature representations. Optionally, the first features of the image to be detected and its adjacent frames (e.g., the previous or next frame of the image to be detected) can be simultaneously input into the inter-frame feature reconstruction module to reconstruct the first features of the image to be detected and its adjacent frames, thereby obtaining the second feature.

[0024] In a specific example, the inter-frame feature reconstruction module is composed of a 3D convolutional network to naturally capture information in the temporal dimension of the video sequence through 3D convolution (Conv3D).

[0025] In another specific example, the inter-frame feature reconstruction module employs a recurrent neural network (RNN), such as LSTM or GRU. In video processing, bidirectional LSTM (BiLSTM) can simultaneously consider information from past and future frames.

[0026] In another specific example, the inter-frame feature reconstruction module is composed of TSM, which is a lightweight temporal modeling method that transmits temporal information by shifting channels in the spatial dimension, making it suitable for real-time applications.

[0027] In another implementation, the inter-frame feature reconstruction module employs an autoencoder structure. During the actual detection of the target image, only the first feature of the target image can be input into the inter-frame feature reconstruction module. This module then reconstructs the second feature from the first feature of the target image. To facilitate the autoencoder-structured inter-frame feature reconstruction module's ability to leverage the high similarity between video frames to reconstruct or enhance the feature representation of the current frame, the inter-frame feature reconstruction loss can be used to train and optimize the target detection network. This loss is calculated using the second feature of the training image and the image features of the next frame. This approach leverages the high similarity of video frame features, using the autoencoder to reconstruct the current frame's features, combined with the next frame's features as supervision. This allows the model to extract information useful for the next frame from the current frame, learn the temporal characteristics of the data, and ultimately achieve data denoising and improve the effectiveness of data extraction.

[0028] During training, training images can be input into the object detection network, where they are processed by the feature extraction module and the inter-frame feature reconstruction module to obtain the second feature of the training images.

[0029] The next frame of the training image is input into the object detection network. At least some units in the feature extraction module of the object detection network process the next frame of the training image to obtain its image features. Specifically, all units in the feature extraction module can process the next frame of the training image to obtain a first feature, which is then used as the image feature of the next frame. Alternatively, some units in the feature extraction module can process the next frame of the training image to obtain its image features. For example, in a feature extraction module containing a backbone network and a neck network, the backbone network can be used to process the next frame of the training image, and the features output by the backbone network can be used as the image features of the next frame.

[0030] After obtaining the second feature of the training image and the image features of the next frame image, the inter-frame feature reconstruction loss can be calculated from these features. The calculation method for the inter-frame feature reconstruction loss is unrestricted; for example, loss functions such as mean squared error, cross-entropy, or Kullback-Leibler divergence can be used. The formula calculates the total loss L of the object detection network, where, It is a basic loss term in object detection networks (such as detection loss). It is the inter-frame feature reconstruction loss of the object detection network. It is the image feature of the next frame of the training image. It is the second feature of the training image.

[0031] Understandably, because the loss function utilizes feature information from the next frame, this method requires training on a video dataset and is not suitable for training datasets with single-frame images. However, during inference, it is not necessary to obtain the features of the next frame in advance, as the loss function does not need to be calculated.

[0032] In this implementation, the autoencoder can be used for data dimensionality reduction, feature learning, and data reconstruction. For example... Figure 2 As shown, an autoencoder can consist of two main parts: an encoder and a decoder. This structure performs dimensionality reduction compression and dimensionality upscaling reconstruction of features, offering lightweight characteristics suitable for online video detection applications. The encoder part of the autoencoder compresses the input data (i.e., the first feature of the image to be detected mentioned above) into a low-dimensional representation; in this process, the encoder learns the latent features or representations of the data. The decoder part reconstructs the original input from the low-dimensional representation, aiming to recover the input data as accurately as possible. In this way, the autoencoder can learn an effective representation of the data. Thus, the autoencoder can effectively compress high-dimensional data to a lower dimension, thereby reducing data complexity while retaining important information, making it an effective method for processing data with less storage space. By rationally designing the data encoding and reconstruction process, functions such as data denoising and missing data supplementation can be achieved.

[0033] Furthermore, this application can employ a convolutional autoencoder structure as the inter-frame feature reconstruction module. Of course, in other embodiments, a traditional fully connected autoencoder structure, a sparse autoencoder structure, or a variational autoencoder structure can be used as the inter-frame feature reconstruction module.

[0034] Taking a convolutional autoencoder as an example, its encoder can include multiple convolutional layers. The function of the convolutional layers is to increase the number of channels of the image features and decrease the feature height and width. In this way, the first feature of the image to be detected can be compressed through multiple convolutional layers. Let's assume that the first feature of the image to be detected is called... Its dimensions are [B, C, H, W], where B, C, H, and W represent the batch size, number of channels, feature height, and feature width, respectively; the image features are processed by several convolutional layers. The dimensions are [B1, C1, H1, W1].

[0035] Optionally, the composition of convolutional layers is unrestricted; for example, it may include convolutional units, or convolutional units, batch normalization units, and activation units. In a specific example, each convolutional layer in the encoder may consist of a 2D convolutional unit, a batch normalization unit, and an activation function unit (such as ReLU, SiLU, or LeakyReLU functions) to enhance the stability of model training by employing batch normalization and activation functions.

[0036] Furthermore, such as Figure 3 As shown, the encoder may also include a reshape layer and / or a fully connected layer. The reshape layer rearranges the elements in the feature tensor, thereby adjusting the tensor's dimension and size. Image features processed by several convolutional layers can be sequentially processed by the reshape layer and the fully connected layer to obtain the third feature of the image to be detected. That is, the encoder can use convolution to perform depth extraction on the input features, then reduce the dimensionality of the depth-extracted feature tensor, and use the fully connected layer for feature mapping to obtain the final low-dimensional feature (i.e., the third feature) output. After the reshape layer adjusts the dimensions to [B1 * H1 * W1, C1], a fully connected layer performs feature mapping to obtain the two-dimensional third feature. Thus, through the combined action of the reshape layer and the fully connected layer, the image features are compressed from four dimensions to two dimensions. Optionally, the fully connected layer consists of a linear layer, a batch normalization operation, and an activation function. In this way, the feature dimension [B1 * H1 * W1, C1] can be adjusted to [B1 * H1 * W1, E] through the linear layer, and finally, after one batch normalization operation and one activation function processing, the third feature is output.

[0037] The decoder of a convolutional autoencoder can include multiple deconvolutional layers. The deconvolutional layers reduce the number of image feature channels while increasing the width and height of the image features. This allows for the decompression of the image features output by the encoder through multiple deconvolutional layers, enabling feature reconstruction through further deconvolutional layers. The feature dimensions output by the decoder are the same as the input dimensions of the encoder part in the autoencoder, which are [B, C, H, W]. This is achieved by ensuring that the output features of the autoencoder-based inter-frame feature reconstruction module maintain the same dimensionality as the input features, such as... Figure 4 As shown, this allows the inter-frame feature reconstruction module to be inserted as a plug-in module into the detection module of a conventional object detection network (i.e., Figure 4Before the detection head, the input features of the detection head are processed, so the inter-frame feature reconstruction module based on the autoencoder of this application has strong adaptability and is applicable to most target detection models based on the backbone network-detection module.

[0038] Optionally, the composition of the deconvolutional layer is not limited; for example, it may include deconvolutional units, or deconvolutional units, batch normalization units, and activation units. In a specific example, each deconvolutional layer in the decoder may consist of a deconvolutional unit, a batch normalization unit, and an activation function unit.

[0039] Furthermore, such as Figure 5 As shown, the decoder can also include a reshape layer. The image features (i.e., the third feature) output by the encoder part of the autoencoder can first be processed by the reshape layer, and then the image features output by the reshape layer are input into the deconvolution layer. The image features output by the reshape layer are then processed by several deconvolution layers in the decoder part to obtain the second feature of the image to be detected. That is, the decoder part of the autoencoder can use the reshape layer to transform the two-dimensional features back into four-dimensional features, and then use the deconvolution layer to upsample the features, restoring the image features to the same size as the encoder input features.

[0040] S13: Use the detection module to perform detection processing on the second feature to obtain the target detection result of the image to be detected.

[0041] After the second feature is obtained by reconstructing the first feature of the image to be detected through the inter-frame feature reconstruction module, the second feature can be detected by the detection module to obtain the target detection result of the image to be detected.

[0042] In this embodiment, after the feature extraction module extracts features from the image to be detected in the video to obtain the first feature, the inter-frame feature reconstruction module can be used to reconstruct the first feature to obtain the second feature. Subsequently, the detection module uses the second feature to detect the target image to obtain the target detection result. This allows the high similarity between video frames to reconstruct or enhance the feature representation of the current frame, enabling the inter-frame feature reconstruction module to reconstruct information in the current frame that is useful for the next frame, thereby achieving feature denoising and improving the detection accuracy of the detection module. This solves the problem of low detection result stability in video target detection scenarios.

[0043] Please see Figure 6 , Figure 6This is a schematic diagram of one embodiment of the electronic device of this application. The electronic device 10 includes a processor 12, which executes instructions to implement the target detection method described above. For detailed implementation processes, please refer to the description of the above embodiment; further details will not be repeated here.

[0044] Processor 12 can also be referred to as a CPU (Central Processing Unit). Processor 12 may be an integrated circuit chip with signal processing capabilities. Processor 12 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or processor 12 may be any conventional processor.

[0045] The electronic device 10 may further include a memory 11 for storing instructions and data required for the processor 12 to run.

[0046] The processor 12 is used to execute instructions to implement the methods provided by any embodiment and any non-conflicting combination of the target detection method and video decompression method of this application.

[0047] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of this application. The computer-readable storage medium 20 in this embodiment stores instruction / program data 21. When executed, this instruction / program data 21 implements the methods provided by any embodiment of the target detection method and video decompression method of this application, as well as any non-conflicting combination thereof. The instruction / program data 21 can be formed into a program file and stored in the storage medium 20 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium 20 includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0048] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0049] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0050] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A target detection method, characterized in that, The target detection method is a method for detecting targets in a video image using a target detection network. The target detection network includes a feature extraction module, an inter-frame feature reconstruction module, and a detection module. The method includes: The feature extraction module is used to extract features from the image to be detected to obtain the first feature; The first feature is reconstructed using the inter-frame feature reconstruction module to obtain the second feature; The detection module is used to detect the second feature to obtain the target detection result of the image to be detected.

2. The target detection method according to claim 1, characterized in that, The target detection network is obtained by optimizing the network using inter-frame feature reconstruction loss, which is calculated using the second feature of the training image and the image feature of the next frame of the training image.

3. The target detection method according to claim 1, characterized in that, The inter-frame feature reconstruction module is an autoencoder structure. The process of reconstructing the first feature using the inter-frame feature reconstruction module to obtain the second feature includes: The encoder portion of the autoencoder structure is used to compress the first feature of the image to be detected into a low-dimensional representation. The original input is reconstructed from the low-dimensional representation using the decoder portion of the autoencoder structure to obtain the second feature of the image to be detected.

4. The target detection method according to claim 3, characterized in that, The encoder portion includes multiple convolutional layers, each convolutional layer comprising a convolutional unit, a batch normalization unit, and an activation function unit connected in sequence. The step of compressing the first feature of the image to be detected into a low-dimensional representation using the encoder portion of the autoencoder structure includes: The first feature of the image to be detected is compressed through the multiple convolutional layers to obtain the low-dimensional representation.

5. The target detection method according to claim 4, characterized in that, The encoder portion further includes a warp layer and a fully connected layer. The step of compressing the first feature of the image to be detected into a low-dimensional representation using the encoder portion of the autoencoder structure includes: The first feature of the image to be detected is compressed by the multiple convolutional layers to obtain four-dimensional features; The four-dimensional features are reduced in dimensionality using the deformation layer to obtain intermediate features; The intermediate features are mapped using the fully connected layer to obtain the two-dimensional low-dimensional representation.

6. The target detection method according to claim 3, characterized in that, The decoder portion includes multiple deconvolutional layers, each including a deconvolution unit, a batch normalization unit, and an activation function unit connected in sequence. The process of reconstructing the original input from the low-dimensional representation using the decoder portion of the autoencoder structure to obtain the second feature of the image to be detected includes: The low-dimensional representation is reconstructed through the multiple deconvolution layers to obtain the second feature of the image to be detected.

7. The target detection method according to claim 6, characterized in that, The decoder portion further includes a warp layer, wherein the reconstruction of the original input from the low-dimensional representation using the decoder portion of the autoencoder structure to obtain the second feature of the image to be detected includes: The deformation layer transforms the two-dimensional low-dimensional representation into four-dimensional features; The four-dimensional features are upsampled and reconstructed by the multiple deconvolution layers to obtain the second feature of the image to be detected.

8. The target detection method according to claim 3, characterized in that, The second feature has the same size as the first feature.

9. An electronic device, characterized in that, The electronic device includes a processor; the processor is configured to execute instructions to implement the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

Images