A target detection method, system, storage medium and electronic device

Abstract

The application provides a target detection method, comprising: acquiring a picture to be detected; inputting the picture to be detected into a backbone network for feature extraction to obtain a multi-scale feature map; inputting the multi-scale feature map into a feature fusion network for feature fusion to obtain an enhanced feature map; inputting the enhanced feature map into a detection head containing a high-frequency channel and a low-frequency channel for detection to output a final result. After obtaining the enhanced feature map, the application performs feature fusion on the input of the enhanced feature map in the detection head containing the high-frequency channel and the low-frequency channel, balances the relationship between the resolution and the number of channels of the feature map in the feature fusion network, and is beneficial to reducing the time consumption of the detection head and improving the detection efficiency of the picture. The application also provides a target detection system, a storage medium and an electronic device, which have the above beneficial effects.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a target detection method, system, storage medium, and electronic device. Background Technology

[0002] With the rapid development of deep learning, object detection algorithms have been widely applied in real-world applications such as autonomous driving, intelligent video surveillance, industrial inspection, and document layout analysis. However, these algorithms suffer from long latency due to complex model structures and high computational demands, making them unsuitable for practical deployment. Therefore, accelerating object detection algorithms has become a crucial problem to solve.

[0003] Current detection algorithms include detection heads, but due to the complexity and diversity of their structures, it is difficult to optimize for speed. In particular, some detection heads contain operators whose specific operations are poorly adapted to hardware or cannot be parallelized, resulting in slow CPU execution speeds. Examples include multi-scale window partitioning, recursion, window expansion, and deformable convolution. These operators are defined as time-consuming operators. The presence of these time-consuming operators leads to long detection head execution times, significantly dragging down the runtime of the detection algorithm. Summary of the Invention

[0004] The purpose of this application is to provide a target detection method, target detection system, storage medium, and electronic device that can improve image detection efficiency while ensuring detection accuracy.

[0005] To address the aforementioned technical problems, this application provides a target detection method, the specific technical solution of which is as follows:

[0006] Obtain the image to be detected;

[0007] The image to be detected is input into the backbone network for feature extraction to obtain a multi-scale feature map.

[0008] The multi-scale feature map is input into a feature fusion network for feature fusion to obtain an enhanced feature map;

[0009] The enhanced feature map is input into a detection head containing a high-frequency channel and a low-frequency channel for detection, and the final result is output. The high-frequency channel is used to extract the local detail features of the enhanced feature map, and the low-frequency channel is used to extract the global structural features of the enhanced feature map.

[0010] Optionally, the step of inputting the image to be detected into the backbone network for feature extraction to obtain a multi-scale feature map includes:

[0011] Determine the backbone network;

[0012] The image to be detected is input into the backbone network, and feature maps of different scales are obtained at different feature extraction stages of the backbone network.

[0013] Optionally, the multi-scale feature map is input into a feature fusion network for feature fusion to obtain an enhanced feature map, including:

[0014] The multi-scale feature map is input into the feature pyramid network for feature fusion to obtain an enhanced feature map.

[0015] Optionally, the enhanced feature maps are input into a detection head containing high-frequency and low-frequency channels for detection, and the final output results include:

[0016] Determine the height, width, and number of channels of the enhanced feature map;

[0017] The enhanced feature map is divided into a high-frequency channel and a low-frequency channel, and the number of feature map channels corresponding to the high-frequency channel is set to a first preset value, the number of feature map channels corresponding to the low-frequency channel is set to a second preset value, and the sum of the first preset value and the second preset value is 1.

[0018] The enhanced feature map is processed using the high-frequency channel and the low-frequency channel respectively, and the processing results of the high-frequency channel and the low-frequency channel are spliced ​​and fused to obtain the final result.

[0019] Optionally, processing the enhanced feature map using the high-frequency channel and the low-frequency channel respectively includes:

[0020] The high-frequency channel is used to process the first type of enhanced feature map in the enhanced feature map, which has a constant resolution and a channel number lower than a preset number.

[0021] The global structural information in the enhanced feature map is extracted using the low-frequency channel.

[0022] Optionally, when extracting global structural information from the enhanced feature map using the low-frequency channel, the method further includes:

[0023] Average pooling is used to reduce the resolution of the enhanced feature map in the low-frequency channel;

[0024] Aggregate local information from the enhanced feature map in the low-frequency channel;

[0025] The resolution of the enhanced feature map is restored to its original size by upsampling.

[0026] Optionally, aggregating local information from the enhancement feature map in the low-frequency channel includes:

[0027] Local information of the enhanced feature map in the low-frequency channel is aggregated in a local window using depthwise disjoint convolutions with residual connections and pointwise convolutions; the local information is aggregated to the center point of the local window.

[0028] This application also provides a target detection system, including:

[0029] The image acquisition module is used to acquire the image to be detected;

[0030] The feature extraction module is used to input the image to be detected into the backbone network for feature extraction to obtain a multi-scale feature map;

[0031] The feature fusion module is used to input the multi-scale feature map into the feature fusion network for feature fusion to obtain an enhanced feature map;

[0032] The target detection module is used to input the enhanced feature map into a detection head containing a high-frequency channel and a low-frequency channel for detection, and output the final result; the high-frequency channel is used to extract the local detail features of the enhanced feature map, and the low-frequency channel is used to extract the global structural features of the enhanced feature map.

[0033] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0034] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described above when it invokes the computer program in the memory.

[0035] This application provides a target detection method, comprising: acquiring an image to be detected; inputting the image to be detected into a backbone network for feature extraction to obtain a multi-scale feature map; inputting the multi-scale feature map into a feature fusion network for feature fusion to obtain an enhanced feature map; inputting the enhanced feature map into a detection head containing a high-frequency channel and a low-frequency channel for detection, and outputting a final result; wherein the high-frequency channel is used to extract local detail features of the enhanced feature map, and the low-frequency channel is used to extract global structural features of the enhanced feature map.

[0036] After obtaining the enhanced feature map, this application performs feature fusion on the enhanced feature map input in a detection head that includes high-frequency and low-frequency channels. By balancing the relationship between the resolution of the feature map and the number of channels in the feature fusion network, it is beneficial to reduce the time consumption of the detection head and improve the detection efficiency of the image.

[0037] This application also provides a target detection system, a storage medium, and an electronic device, which have the aforementioned beneficial effects, and will not be elaborated here. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a target detection method provided in an embodiment of this application;

[0040] Figure 2 This is a diagram of a high- and low-frequency dual-channel network structure provided in an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of the structure of the local feature aggregation module provided in the embodiments of this application;

[0042] Figure 4 This is a schematic diagram of a target detection system provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] Please refer to Figure 1 , Figure 1 A flowchart of a target detection method provided in an embodiment of this application, the method comprising:

[0045] S101: Obtain the image to be detected;

[0046] This step aims to obtain the image to be detected. There are no restrictions on how to obtain the image; it can be any image that needs to be detected, and there are no restrictions on image resolution, etc.

[0047] S102: Input the image to be detected into the backbone network for feature extraction to obtain a multi-scale feature map;

[0048] This step requires inputting the image to be detected into the backbone network for feature extraction, thereby obtaining multi-scale feature maps. Specifically, the backbone network can be determined first, and then the image to be detected can be input into the backbone network. Different scale feature maps are obtained at different feature extraction stages of the backbone network. The backbone network can be ResNet, MobileNetV2, ShuffleNet, or Vision Transformer. This invention uses the highly efficient MobileNetV2 as an example, using MobileNetV2 as the backbone network for object detection to extract features, and finally outputting four different scale feature maps at different stages of the network. It should be noted that as the network deepens, the extracted features become more high-order, the feature map resolution decreases, and the number of channels increases.

[0049] S103: Input the multi-scale feature map into the feature fusion network for feature fusion to obtain an enhanced feature map;

[0050] This step requires feature fusion. Specifically, the multi-scale feature maps can be input into a feature pyramid network for feature fusion to obtain enhanced feature maps. The feature pyramid network then fuses high-resolution low-order features with low-resolution high-order features to obtain enhanced feature maps. This ensures that the enhanced feature maps contain both image detail features and global structural information. The number of channels in the enhanced multi-scale feature maps is uniformly set to c.

[0051] S104: Input the enhanced feature map into a detection head containing a high-frequency channel and a low-frequency channel respectively for detection, and output the final result; the high-frequency channel is used to extract the local detail features of the enhanced feature map, and the low-frequency channel is used to extract the global structural features of the enhanced feature map.

[0052] The detection head in this step includes a high-frequency channel and a low-frequency channel. The high-frequency channel is used to process the first type of enhanced feature map with unchanged resolution and fewer than the preset number of channels in the enhanced feature map, and the low-frequency channel is used to extract global structural information from the enhanced feature map.

[0053] High-frequency channels are responsible for processing feature maps with constant resolution and a smaller number of channels, capturing local image details. Because the number of feature map channels is reduced, the computational cost of subsequent deformation convolutions used for feature extraction is also reduced, thus accelerating the process. Low-frequency channels are responsible for extracting global structural information from the image and have lower requirements for local details. Since there is significant redundancy between adjacent pixels in the image, as a preferred approach, the resolution of low-frequency channel feature maps can be reduced using average pooling, for example, to one-quarter of their original size, greatly reducing the computational cost of subsequent processing.

[0054] See Figure 2 , Figure 2 The high- and low-frequency dual-channel network structure diagram provided in this application embodiment, as a preferred execution method, can be implemented using the following process:

[0055] Step 1: Determine the height, width, and number of channels of the enhanced feature map;

[0056] The second step is to divide the enhanced feature map into a high-frequency channel and a low-frequency channel, and set the number of feature map channels corresponding to the high-frequency channel to a first preset value, the number of feature map channels corresponding to the low-frequency channel to a second preset value, and the sum of the first preset value and the second preset value is 1.

[0057] The third step involves processing the enhanced feature map using the high-frequency channel and the low-frequency channel respectively, and then splicing and fusing the processing results of the high-frequency channel and the low-frequency channel to obtain the final result.

[0058] For ease of explanation, h and w represent the height and width of the feature map input to the detection head, respectively, and c represents the number of feature map channels. This application divides the feature map into high-frequency and low-frequency channels using a convolution with a kernel of 1. A weight parameter λ is introduced to balance the ratio of the two channel numbers. The number of channels in the high-frequency channel is set to λ*c, and the number of channels in the low-frequency channel is set to (1-λ)*c. A larger value of λ results in more significant acceleration and can be adjusted based on experimental results.

[0059] As a preferred execution method, a local feature aggregation (LA) module can be introduced to aggregate local information of the image, mitigating the negative impact of resolution reduction caused by average pooling on features. The structure of the local feature aggregation module is as follows: Figure 3 As shown, Figure 3 This is a schematic diagram of the local feature aggregation module provided in an embodiment of this application. The module aggregates information in a local window of size k×k using depth-wise convolution and point-wise convolution with residual connections. Here, the value of k is not limited, for example, k=3.

[0060] Furthermore, residual connections can be introduced into the local feature aggregation module to fuse high-order and low-order features, enriching the information in the feature map and further improving detection performance. See [link to relevant documentation]. Figure 3 The specific process is as follows:

[0061] The first step is to input the pooled feature map into the deformable convolution for feature extraction. Since the resolution and number of channels are greatly reduced, the amount of computation is greatly reduced, thereby achieving the goal of acceleration.

[0062] The second step is to facilitate the fusion of low-frequency and high-frequency channel features. The feature map after deformation convolution needs to be upsampled and then passed through a linear mapping layer to obtain a feature map with width w, height h and number of channels (1-λ)*c.

[0063] The third step involves concatenating the processed high- and low-frequency features and outputting the results to subsequent convolutional layers for classification and regression. Because the high-frequency channel reduces the number of channels in the feature map and the low-frequency channel reduces its resolution, the overall model framework is accelerated on the CPU. Comprehensive benchmark tests demonstrate that the high- and low-frequency dual-channel network structure outperforms existing model structures in terms of performance, FLOPs, latency, and memory consumption for detectors containing time-consuming operators (e.g., deformation convolution), achieving approximately a 3x speedup.

[0064] This step applies the detection head to the enhanced feature maps output from the previous step for target classification and regression, respectively, and outputs the final result. This final result refers to the detection result of the image to be detected; it is not specifically limited here and may include the category and location information of the detected targets.

[0065] In this embodiment, after obtaining the enhanced feature map, feature fusion is performed on the input enhanced feature map in a detection head containing high-frequency and low-frequency channels. By balancing the relationship between the resolution of the feature map and the number of channels in the feature fusion network, the time consumption of the detection head is reduced, and the detection efficiency of the image is improved. The high-frequency and low-frequency channels are responsible for extracting features at different levels of the image, respectively. The high-frequency channel is responsible for extracting local structural information of the image, requiring a higher resolution feature map, but not a high number of feature map channels. The low-frequency channel is responsible for extracting global structural information of the image, and does not require a high resolution feature map. Since there is redundancy between adjacent pixels in the image, the resolution of the feature map can be further reduced by downsampling to achieve the purpose of acceleration. When using downsampling, a local feature aggregation module containing residual connections can be introduced into the low-frequency channel. Local features of the image are aggregated through depthwise separable convolution and pointwise convolution, which alleviates the feature loss caused by average pooling downsampling.

[0066] By introducing a feature map channel number balancing parameter λ, the ratio of high-frequency and low-frequency feature map channel numbers can be controlled while keeping the total number of dual-channel feature map channels constant, thereby controlling the computational load of the model.

[0067] The following describes a target detection system provided by an embodiment of this application. The target detection system described below can be referred to in correspondence with the target detection method described above.

[0068] See Figure 4 , Figure 4This application provides a schematic diagram of a target detection system structure according to an embodiment of the present application. The present application also provides a target detection system, including:

[0069] The image acquisition module is used to acquire the image to be detected;

[0070] The feature extraction module is used to input the image to be detected into the backbone network for feature extraction to obtain a multi-scale feature map;

[0071] The feature fusion module is used to input the multi-scale feature map into the feature fusion network for feature fusion to obtain an enhanced feature map;

[0072] The target detection module is used to input the enhanced feature map into a detection head containing a high-frequency channel and a low-frequency channel for detection, and output the final result; the high-frequency channel is used to extract the local detail features of the enhanced feature map, and the low-frequency channel is used to extract the global structural features of the enhanced feature map.

[0073] Based on the above embodiments, as a preferred embodiment, the feature extraction module includes:

[0074] Backbone network determination unit, used to determine the backbone network;

[0075] The feature extraction unit is used to input the image to be detected into the backbone network and obtain feature maps of different scales at different feature extraction stages of the backbone network.

[0076] Based on the above embodiments, as a preferred embodiment, the feature fusion module is used to input the multi-scale feature map into the feature pyramid network for feature fusion to obtain an enhanced feature map.

[0077] Based on the above embodiments, as a preferred embodiment, the target detection module includes:

[0078] The feature map parameter determination unit is used to determine the height, width, and number of channels of the enhanced feature map;

[0079] The channel setting unit is used to divide the enhanced feature map into a high-frequency channel and a low-frequency channel, and set the number of feature map channels corresponding to the high-frequency channel to a first preset value, the number of feature map channels corresponding to the low-frequency channel to a second preset value, and the sum of the first preset value and the second preset value is 1.

[0080] The feature map processing unit is used to process the enhanced feature map using the high-frequency channel and the low-frequency channel respectively, and to splice and fuse the processing results of the high-frequency channel and the low-frequency channel to obtain the final result.

[0081] Based on the above embodiments, as a preferred embodiment, the feature map processing unit is a unit for performing the following steps:

[0082] The high-frequency channel is used to process the first type of enhanced feature map in the enhanced feature map, which has a constant resolution and a channel number lower than a preset number.

[0083] The global structural information in the enhanced feature map is extracted using the low-frequency channel.

[0084] Based on the above embodiments, as a preferred embodiment, the feature map processing unit further includes:

[0085] A pooling unit is used to reduce the resolution of the enhanced feature map in the low-frequency channel using average pooling; aggregate local information of the enhanced feature map in the low-frequency channel; and restore the resolution of the enhanced feature map to its original size through upsampling.

[0086] Based on the above embodiments, as a preferred embodiment, the pooling unit includes:

[0087] An aggregation subunit is used to aggregate local information of the enhanced feature map in the low-frequency channel in a local window using a depthwise disjoint convolution and a pointwise convolution that includes residual connections; the local information is aggregated to the center point of the local window.

[0088] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0089] This application also provides an electronic device that may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the electronic device may also include various network interfaces, power supplies, and other components.

[0090] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. As the system provided in the embodiments corresponds to the method provided in the embodiments, the description is relatively simple; relevant parts can be found in the method section.

[0091] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0092] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A target detection method, characterized in that, include: Obtain the image to be detected; The image to be detected is input into the backbone network for feature extraction to obtain a multi-scale feature map. The multi-scale feature map is input into a feature fusion network for feature fusion to obtain an enhanced feature map; The enhanced feature map is input into a detection head containing a high-frequency channel and a low-frequency channel for detection, and the final result is output. The high-frequency channel is used to extract local detail features of the enhanced feature map, and the low-frequency channel is used to extract global structural features of the enhanced feature map. The enhanced feature maps are input into a detection head containing both high-frequency and low-frequency channels for detection, and the final output results include: Determine the height, width, and number of channels of the enhanced feature map; The enhanced feature map is divided into a high-frequency channel and a low-frequency channel, and the number of feature map channels corresponding to the high-frequency channel is set to λ*c, and the number of feature map channels corresponding to the low-frequency channel is (1-λ)*c, where c represents the number of feature map channels, and the weight parameter λ is used to balance the ratio of the two channel numbers. The enhanced feature map is processed using the high-frequency channel and the low-frequency channel respectively, and the processing results of the high-frequency channel and the low-frequency channel are spliced ​​and fused to obtain the final result; The processing of the enhanced feature map using the high-frequency channel and the low-frequency channel respectively includes: The high-frequency channel is used to process the first type of enhanced feature map in the enhanced feature map, which has a constant resolution and a channel number lower than a preset number. The global structural information in the enhanced feature map is extracted using the low-frequency channel; Average pooling is used to reduce the resolution of the enhanced feature map in the low-frequency channel; Aggregate local information from the enhanced feature map in the low-frequency channel; The resolution of the enhanced feature map is restored to its original size by upsampling.

2. The target detection method according to claim 1, characterized in that, The step of inputting the image to be detected into the backbone network for feature extraction to obtain a multi-scale feature map includes: Determine the backbone network; The image to be detected is input into the backbone network, and feature maps of different scales are obtained at different feature extraction stages of the backbone network.

3. The target detection method according to claim 1, characterized in that, The multi-scale feature maps are input into a feature fusion network for feature fusion to obtain enhanced feature maps, including: The multi-scale feature map is input into the feature pyramid network for feature fusion to obtain an enhanced feature map.

4. The target detection method according to claim 1, characterized in that, Aggregating local information from the enhanced feature map in the low-frequency channel includes: Local information of the enhanced feature map in the low-frequency channel is aggregated in a local window using depthwise disjoint convolutions with residual connections and pointwise convolutions; the local information is aggregated to the center point of the local window.

5. A target detection system, characterized in that, include: The image acquisition module is used to acquire the image to be detected; The feature extraction module is used to input the image to be detected into the backbone network for feature extraction to obtain a multi-scale feature map; The feature fusion module is used to input the multi-scale feature map into the feature fusion network for feature fusion to obtain an enhanced feature map; the feature fusion network includes a high-frequency channel and a low-frequency channel. The target detection module is used to input the enhanced feature map into the detection head containing high-frequency channels and low-frequency channels respectively for detection, and output the final result; The target detection module includes: The feature map parameter determination unit is used to determine the height, width, and number of channels of the enhanced feature map; The channel setting unit is used to divide the enhanced feature map into high-frequency channels and low-frequency channels, and set the number of feature map channels corresponding to the high-frequency channels to λ*c, and the number of feature map channels corresponding to the low-frequency channels to (1-λ)*c, where c represents the number of feature map channels, and the weight parameter λ is used to balance the ratio of the two channel numbers. The feature map processing unit is used to process the enhanced feature map using the high-frequency channel and the low-frequency channel respectively, and to splice and fuse the processing results of the high-frequency channel and the low-frequency channel to obtain the final result; The feature map processing unit is a unit used to perform the following steps: The high-frequency channel is used to process the first type of enhanced feature map with unchanged resolution and fewer than a preset number of channels in the enhanced feature map; the low-frequency channel is used to extract global structural information from the enhanced feature map; In addition, the feature map processing unit also includes: A pooling unit is used to reduce the resolution of the enhanced feature map in the low-frequency channel using average pooling; aggregate local information of the enhanced feature map in the low-frequency channel; and restore the resolution of the enhanced feature map to its original size through upsampling.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the target detection method as described in any one of claims 1-4.

7. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the target detection method as described in any one of claims 1-4.

Images