Complex scene small target detection device, method and system, and storage medium

Abstract

The invention discloses a complex scene small target detection device, method and system, and a storage medium, and the device comprises a self-adaptive enhancement module which is used for improving the focusing and extraction capability of key features of different types of targets in a complex scene, and comprises the steps: carrying out the self-adaptive enhancement preprocessing of a low-quality image, and enhancing the key feature extraction of different types of targets; the feature extraction module is used for adaptively mining a multi-scale feature map representing key features of the small target from the enhanced image; the feature fusion module is used for fusing a shallow-layer feature map containing target detail information and a deep-layer feature map containing category semantic information according to the multi-scale features and improving the sensitivity to complex and irregular small target features; and the prediction module is used for carrying out target classification and positioning according to the fusion feature map obtained from the feature fusion module. By adopting the technical scheme of the invention, high-precision identification and positioning of a small target are realized, and meanwhile, the adaptive capability of multiple complex scenes is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular, relates to a device and method, a system, and a storage medium for detecting small targets in complex scenes. Background Art

[0002] With the rapid development of computer vision technology, target detection has become a core task in this field. Target detection not only needs to locate all targets in the image, but also needs to identify the category of each target. In recent years, thanks to the progress of deep learning technology, target detection algorithms have made significant breakthroughs in accuracy and speed. However, in practical applications, we often face the challenge of complex small target detection. Small targets usually refer to targets that occupy a small pixel area in the image, such as vehicles in remote sensing images, pedestrians in surveillance videos or animals in field images, surface defects of industrial products, and various complex targets in coal mines. Due to the uneven scale of small targets, irregular visualization features, and small area occupied in the image, when the environment is complex, the lighting conditions are poor, and the image quality is low, it is often accompanied by complex background interference, target occlusion and other problems. In such cases, small target detection is more difficult than general target detection.

[0003] The algorithms currently used for target detection can be mainly divided into two categories: single-stage and two-stage target detection algorithms. Single-stage target detection algorithms usually have the advantages of fast speed, simple and easy deployment and high real-time performance. The algorithm can simultaneously realize the regression of the prediction box and the classification prediction task, and realize end-to-end target detection. The classic single-stage target detection algorithms mainly include: YOLO (You Only Look Once) series algorithms and SSD (Single Shot Detector) series algorithms. The two-stage target detection algorithm has two stages: candidate box generation and classification fine adjustment. It can achieve higher accuracy in target positioning and classification, and is easier to handle multi-scale targets. However, the two-stage method often has the disadvantages of high complexity, difficulty in deployment and unsuitability for high real-time applications. The classic two-stage target detection algorithms mainly include: R-CNN (Region with CNNfeature) series algorithms and Faster R-CNN algorithms. In view of the detection and positioning needs of complex small targets in many current industrial scenarios, there are higher requirements for target detection algorithms to be highly accurate, high-speed, and easy to deploy in devices with limited computing resources. Therefore, single-stage target detection algorithms have more advantages than two-stage target detection algorithms. In recent years, with the continuous development of the YOLO network, many researchers have designed target detection models based on the YOLO network for different scenarios. However, the detection methods for complex small targets not only face the challenges of large algorithm model parameters, long inference time, and unsatisfactory false detection and missed detection rates, but also face challenges such as susceptibility to environmental interference and difficulty in extracting target features due to the uneven scale of small targets, unclear and irregular visualization features, etc. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a small target detection device and method, system and storage medium that can adapt to different complex scenes.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A small target detection device in a complex scene, comprising:

[0007] Adaptive enhancement module, used to perform adaptive enhancement preprocessing on low-quality images containing complex small targets and to perform feature extraction and enhancement on key features of different types of targets;

[0008] The feature extraction module is used to adaptively mine multi-scale feature maps that characterize the key features of small targets from the enhanced image;

[0009] A feature fusion module is used to fuse the shallow feature map containing target detail information and the deep feature map containing category semantic information based on the multi-scale features obtained from the feature extraction module;

[0010] The prediction module is used to classify and locate the target based on the fused feature map obtained from the feature fusion module.

[0011] As a preferred method, the feature extraction module is constructed based on the designed DSC_Attention module, two-dimensional convolution module Conv, dynamic snake convolution module DSConv, Attention module and SPP module. Among them, the embedding of the attention mechanism in DSC_Attention and Attention adopts the optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, and the initial default embedded attention mechanism module is the CPCA attention mechanism module.

[0012] Preferably, the adaptive enhancement module includes: a low-quality image adaptive enhancement network based on deep reinforcement learning and a feature extraction adaptive enhancement module based on a heat map and a DDQN multi-attention mechanism adaptive embedding network.

[0013] Preferably, the feature fusion module comprises: 3 DSConv modules, 6 DSCSP modules, 6 SDFM modules and 3 Upsample upsampling modules; wherein, the bottom-up fusion path in the feature fusion module includes: 3 SDFM modules are placed in the L13, L16 and L19 layers respectively; 3 DSCSP modules are placed in the L14, L17 and L20 layers after the 3 SDFM modules respectively; L12, L15 and L18 are Upsample upsampling modules; the top-down path includes 3 groups of identical module structures: L21-L23 layers are constructed by 1 DSConv module, 1 SDFM module and 1 DSCSP module, and L24-L26 layers and L27-L29 layers are also constructed in the same way.

[0014] The present invention also provides a method for detecting small targets in complex scenes, comprising:

[0015] Adaptive enhancement module is used to perform adaptive enhancement preprocessing on low-quality images containing complex small targets and to extract and enhance key features of different types of targets;

[0016] The feature extraction module adaptively mines a multi-scale feature map representing the key features of small targets from the enhanced image.

[0017] The feature fusion module fuses the shallow feature map containing target detail information and the deep feature map containing category semantic information based on the multi-scale features obtained from the feature extraction module.

[0018] The prediction module performs target classification and positioning based on the fused feature map obtained from the feature fusion module.

[0019] As a preferred method, the feature extraction module is constructed based on the designed DSC_Attention module, two-dimensional convolution module Conv, dynamic snake convolution module DSConv, Attention module and SPP module. Among them, the embedding of the attention mechanism in DSC_Attention and Attention adopts the optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, and the initial default embedded attention mechanism module is the CPCA attention mechanism module.

[0020] Preferably, the adaptive enhancement module includes: a low-quality image adaptive enhancement network based on deep reinforcement learning and a feature extraction adaptive enhancement module based on a heat map and a DDQN multi-attention mechanism adaptive embedding network.

[0021] Preferably, the feature fusion module comprises: 3 DSConv modules, 6 DSCSP modules, 6 SDFM modules and 3 Upsample upsampling modules; wherein, the bottom-up fusion path in the feature fusion module includes: 3 SDFM modules are placed in the L13, L16 and L19 layers respectively; 3 DSCSP modules are placed in the L14, L17 and L20 layers after the 3 SDFM modules respectively; L12, L15 and L18 are upsampling modules; the top-down path includes 3 groups of identical module structures: L21-L23 layers are constructed by 1 DSConv module, 1 SDFM module and 1 DSCSP module, and L24-L26 layers and L27-L29 layers are also constructed in the same way.

[0022] An embodiment of the present invention further provides a complex scene small target detection system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a complex scene small target detection method when executed by the processor.

[0023] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes the method for detecting small targets in complex scenes when running.

[0024] Based on the research on deep reinforcement learning, dynamic snake-shaped convolutional neural network, attention mechanism embedding, efficient fusion of multi-scale features and optimization of small target detection layer, the present invention designs a complex scene small target detection method based on reinforcement learning adaptive enhancement, which can achieve high-precision recognition and positioning of small targets while having the adaptability to multiple complex scenes. The present invention has the following technical effects:

[0025] (1) In order to effectively extract the key features of small-scale and irregularly distributed small targets in complex environment images and improve the positioning accuracy and detection sensitivity of small targets, a complex small target feature extraction module (DSAM) based on dynamic snake convolution and attention mechanism is proposed. This module introduces dynamic snake convolution to expand the receptive field and enhance the network's ability to extract complex and irregular target features; by pre-embedding the CPCA attention mechanism module, the feature extraction network's ability to focus on the detailed features of small targets is improved, thereby suppressing the interference of complex backgrounds and improving the detection sensitivity of targets.

[0026] (2) In order to achieve the goal of adaptively adapting to low-quality images collected in different environments during target detection and improve the detection accuracy and generalization performance of the detection model, a complex scene target image adaptive enhancement module based on DDQN is designed. Based on the four image enhancement processing sub-modules (image denoising, dehazing, brightness enhancement and super-resolution), this module combines the reinforcement learning optimization model built based on DDQN to adaptively select the optimal image enhancement method combination to enhance the input low-quality images.

[0027] (3) In order to realize the feature extraction module's ability to adapt to different types of targets in complex environments, a feature extraction network multi-attention mechanism adaptive embedding module based on heat map and DDQN is proposed. This module adopts the optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, combined with five alternative attention mechanism modules (channel attention mechanism, spatial attention mechanism, coordinate attention mechanism, CBAM attention mechanism and EMA attention mechanism module), to achieve adaptive adjustment of the attention mechanism in the feature extraction module when dealing with different types of targets in other complex environments, so as to achieve more ideal target key feature focus and extraction performance.

[0028] (4) In order to efficiently fuse shallow features containing detail information and deep features containing semantic information with the multi-scale features obtained from the feature extraction module, a feature fusion module, MBiFPN, is proposed that integrates dynamic snake convolution and multi-attention mechanism. Based on the BiFPN structure, this module introduces dynamic snake convolution, semantic and detail information fusion module embedded with coordinate attention and spatial attention mechanism, which can enhance the multi-scale feature fusion performance and use multi-attention mechanism to improve the model's focusing ability and detection sensitivity for complex small targets, thereby reducing false detection and missed detection of targets.

[0029] (5) In order to further improve the positioning accuracy of the target detection model for complex small targets and accelerate the convergence of the loss function, four new independent decoupled detection heads DSCSHead are designed to perform the positioning regression and recognition classification tasks of the target detection model. In DSCSHead, dynamic snake convolution is introduced to improve the positioning accuracy of complex small targets, and SIoU is used to improve the regression loss function to accelerate model convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 This is a schematic diagram of the structure of a small target detection device in a complex scene according to an embodiment of the present invention;

[0032] Figure 2 It is the internal structure of the DSC_Attention module;

[0033] Figure 3 This is the DSAM module structure diagram;

[0034] Figure 4 It is the LIAENDRL structure;

[0035] Figure 5 Schematic diagram of the action space for learning the optimal enhancement strategy for low-quality images based on DDQN;

[0036] Figure 6 Schematic diagram of historical action vector;

[0037] Figure 7 It is a MAMHD structure;

[0038] Figure 8 Schematic diagram of the action space for learning the optimal attention mechanism embedding strategy based on DDQN;

[0039] Fig. 9 Schematic diagram of historical action vector;

[0040] Fig.10 This is the SDFM module structure diagram;

[0041] Fig.11 This is the DSCSP module structure diagram;

[0042] Fig.12 This is the DSCSHead module structure diagram;

[0043] Fig.13 Schematic diagram of parameters related to SIoU loss function. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Embodiment 1:

[0047] like Figure 1 As shown, an embodiment of the present invention provides a small target detection device in a complex scene, comprising:

[0048] Adaptive enhancement module, used to perform adaptive enhancement preprocessing on low-quality images containing complex small targets and to perform feature extraction and enhancement on key features of different types of targets;

[0049] The feature extraction module DSAM is used to adaptively mine multi-scale feature maps that represent the key features of small targets from the enhanced image;

[0050] The feature fusion module MBiFPN is used to fuse the shallow feature map containing target detail information and the deep feature map containing category semantic information based on the multi-scale features obtained from the feature extraction module;

[0051] The prediction module is used to classify and locate the target based on the fused feature map obtained from the feature fusion module.

[0052] As an implementation method of the embodiment of the present invention, the working process of the adaptive enhancement module is as follows:

[0053] In order to improve the adaptability of target detection models to different types of targets in different complex scenes, the adaptive ability of detection models is improved from two aspects: image enhancement preprocessing and feature extraction. In terms of image preprocessing, a complex scene target image adaptive enhancement preprocessing module is designed to perform adaptive enhancement preprocessing on low-quality images in different complex scenes. In terms of feature extraction enhancement, based on the optimal attention mechanism adaptive embedding strategy of deep reinforcement learning, a feature extraction adaptive enhancement module based on heat map and DDQN is proposed. The attention mechanism that is more conducive to the extraction of key features of different types of targets is adaptively selected and embedded into the feature extraction module, thereby improving the feature extraction module's ability to extract key features of different types of targets.

[0054] Complex scene target image adaptive enhancement preprocessing module

[0055] In many cases of complex environments, poor lighting conditions, and limited performance of image acquisition equipment, the acquired target images will face problems such as poor quality, blurred targets, low resolution, and unclear visual features, which will seriously affect the accuracy of target detection. In this regard, in order to adapt to the target detection of low-quality images collected in different environments and improve the detection accuracy and the generalization performance of the detection model, a low-quality image preprocessing module is designed. The low-quality image adaptive enhancement network based on deep reinforcement learning (LIAENDRL) is used to adaptively adopt image enhancement strategies that are conducive to improving detection accuracy for low-quality images in different environments, convert low-quality images into high-quality images, and finally improve the model's target feature extraction effect on low-quality images.

[0056] The structure of the image enhancement preprocessing module LIAENDRL designed in the embodiment of the present invention is as follows: Figure 4 The specific description is as follows:

[0057] (1) Image enhancement method. In order to adapt to the enhancement of low-quality images in different scenarios, four image enhancement processing submodules are used for image denoising, defogging, brightness enhancement and super-resolution. Among them, the image denoising adopts the bilateral filtering algorithm to smooth the image while maintaining the target edge information; the removal processing adopts the dark channel prior defogging algorithm; the brightness enhancement adopts the URetinex-Net algorithm; and the image super-resolution adopts the Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN).

[0058] (2) Adaptive optimization of low-quality image enhancement strategies based on DDQN. When faced with low-quality images, experienced experts and researchers usually manually select and adjust image enhancement algorithms and parameters according to their application scenarios. However, this manual method of adjusting image enhancement algorithms and parameters makes it difficult for target detection models to adapt to different usage scenarios. For example, images in scenes such as heavy fog and smoke need to focus on dehazing, and images in scenes with low light or uneven light distribution need to focus on brightness enhancement. For low-resolution images, super-resolution image reconstruction is required to improve the target feature extraction effect. In this regard, an embodiment of the present invention uses a deep reinforcement learning method DDQN to learn image enhancement strategies to replace the manual selection of image enhancement algorithms. Figure 4 As shown in the figure, low-quality images are input into DDQN, and the strategy for image enhancement processing is output. The optimal algorithm combination is selected from the four image enhancement processing methods to enhance the low-quality images and obtain high-quality images that can effectively improve the accuracy of target detection.

[0059] The network structure of DDQN consists of three parts. The first part is the feature extraction layer, which is composed of a three-layer CNN network. The second part is two sub-networks composed of fully connected layers. One sub-network is the state value function network V(s; θ, β), where s represents the state, θ represents the CNN network parameters, and β is the parameter of the state value function network V. The other sub-network is the advantage function network A(s, a; θ, α) for predicting each action, where s represents the state, a represents the action to be performed, θ represents the CNN network parameters, and α is the parameter of the advantage function network A. The third part is to combine the results of the two sub-networks V(s; θ, β) and A(s, a; θ, α) of the second part to obtain the final state-action value Q, which is expressed as:

[0060] Q(s,a;θ,α,β)=V(s;θ,β)+A(s,a;θ,α) (15)

[0061] For the training and learning of the optimal enhancement strategy for low-quality images based on DDQN, through continuous contact with the environment, the action with the largest value Q is selected as the next step according to the current strategy of the model. In this process, the specific action reward value given by the environment is used to evaluate the effect of the action execution. Through iterative optimization, the learning process is finally terminated under certain conditions and the optimal strategy is output.

[0062] For the optimal enhancement strategy learning model for low-quality images based on DDQN designed in the embodiment of the present invention, the specific construction of the action space, state space, reward function and training optimization process is as follows:

[0063] ① Action space. The action space records all the actions that the DDQN agent can take in the environment of low-quality image processing. The embodiment of the present invention designs 4 image processing modules, each of which has 2 execution actions: perform the processing of the module and no operation (do not perform the processing of the module). Therefore, a total of 16 actions can be executed. The schematic diagram of the action space is as follows Figure 5 shown. Figure 6 The figure shows the history of the last four selected actions. The dimension of the vector is 4×4, denoted as h, and the initial value is 0. Each time a new vector h0 is constructed for the currently selected action, the dimension of this new vector is 1×4, and the action expressed by the vector is the same as Figure 5 Whenever h is updated, it first scrolls down 1 step in the row dimension, and then assigns h0 to the first row of the current h. When the agent chooses to terminate the action, it means that the current state is the best according to the strategy, and no action is taken. This indicates that the action decision iteration process will end, and the current action vector is output as the final result.

[0064] ② State space. The state represents the information of the current environment and strongly affects the model's decision on the next action. Therefore, the state space should contain enough comprehensive information. In the task of adaptive enhancement of low-quality images, the low-quality image I to be enhanced is input, the execution action a obtained from the DDQN-based low-quality image optimal enhancement strategy learning model, and the corresponding reward signal r after the execution of the action.

[0065] ③ Reward function. The reward function is the key to guiding the DDQN agent to find the best action selection strategy. In the embodiment of the present invention, the agent adjusts the image enhancement processing strategy through a series of action sequences, and finally determines the optimal image enhancement module combination. Therefore, it is hoped that in the training process, the images enhanced according to the strategy will be gradually more conducive to improving the target detection accuracy. At this time, the images enhanced according to the strategy are input into the pre-trained target detection model, and the target detection accuracy obtained is used as the reward function to measure the effectiveness of the agent's action selection. Specifically, the target detection accuracy obtained by the image enhanced by the current strategy is greater than the target detection accuracy obtained by the image enhanced by the previous strategy, indicating that the strategic action at this time is more conducive to improving the target detection accuracy and should be rewarded; otherwise, it should be punished. By constructing an appropriate objective function, the problem can be transformed into an objective function optimization problem that maximizes the reward, so that the agent can make decisions that meet expectations. The reward function for each action adopted in the embodiment of the present invention is:

[0066] R t (s t ,s t+1 )=sign(Acc(O t+1 ,M pre )-Acc(Ot ,M pre )) (16)

[0067] In the formula, O t+1 represents the enhanced image after this action is performed, O t represents the enhanced image after the last action is performed, M pre Acc(O) is a target detection model pre-trained with low-quality images without enhancement. t+1 ,M pre ) means O t+1 Enter M pre The target detection accuracy obtained after that. The sign function takes the value of 1 or -1 depending on whether the value of the internal expression is greater than 0. In order to minimize the number of selections, different reward functions are required for the termination action, as shown below:

[0068]

[0069] Where s1 represents the initial state, the low-quality image has not been enhanced, and I is the low-quality image. When the target detection accuracy of the enhanced image after this action is greater than that of the original low-quality image by τ, a reward η is obtained, otherwise the reward is -η.

[0070] ④ Training optimization. The training process of DDQN is summarized as follows: a training set consisting of 100 low-quality images with low illumination, noise, blur and low resolution is selected, and then 100 high-quality images are selected to form the training data. In addition, the "exploration and utilization" method is introduced, and the target network and the experience replay pool are combined to improve the training efficiency and model performance. First, by establishing an experience replay pool R for model training, the state, action, reward and next state of each time step are recorded as an experience and stored. When the experience pool is a mixture of past and recent experiences, the experience samples randomly sampled from R have little correlation, and these experiences can be reused. Then, the training network Q(s,a;θ) is used to guide action decisions, and the target network Q - (s′, a′; θ′) is used to estimate the target Q value; the training network has the same structure as the target network. To reduce the amount of calculation, the target network directly obtains the parameters of the current training network when it needs to be updated; the update process of the training network is completed by minimizing the loss function of formula (3-12):

[0071] L(s,a|θ)=(yQ(s,a|θ)) 2 (18)

[0072]

[0073] In the formula, y represents the reward value. When it is a non-terminal action, y is the superposition of the discounted value of the immediate reward and the future reward; when it is a terminal action, y is equal to the immediate reward. γ is the discount factor, which is used to determine the proportion of future rewards in the cumulative reward. Its value range is 0 to 1. In order to learn and gain more experience, the "exploration and utilization" method is used, and the ε-greedy strategy is adopted to determine the action, execute the optimal action with a probability of 1-ε, and execute other actions with a probability of ε. The formula is as follows:

[0074]

[0075] Feature extraction adaptive enhancement module

[0076] In order to improve the target detection network's ability to focus on and extract key features of different types of targets (scale, shape, etc.), a feature extraction adaptive enhancement module is designed. The Multi-attention mechanism adaptive embedding network for feature extraction module based on heatmaps and DDQN (MAMHD) can adaptively select the attention mechanism that is more conducive to key feature extraction and embed it into the feature extraction network when facing different types of targets, thereby improving the feature extraction module's ability to extract key features of different types of targets.

[0077] The MAMHD structure designed in the embodiment of the present invention is as follows Figure 7 The specific description is as follows:

[0078] (1) Multiple attention mechanism modules. In order to adapt to the key feature focusing and extraction of different types of targets in complex scenes, the default attention mechanism module embedded in the feature extraction module is the CPCA attention mechanism module, and the modules involved include: DSC_Attention module and Attention module; in addition, a variety of different types of attention mechanism modules are used as candidate modules for embedding in the feature extraction module, including channel attention mechanism module, spatial attention mechanism module, coordinate attention mechanism module, CBAM attention mechanism module, and EMA attention mechanism module.

[0079] (2) Optimization of the attention mechanism embedding strategy based on DDQN. Since there are often obvious differences in the targets to be detected in complex scenes, such as uneven scales and diverse shapes. Therefore, in the feature extraction process, different types of targets need to focus on different information. When facing different types of target images, the selection of a suitable attention mechanism module can be more conducive to the focus and extraction of key features, thereby improving the detection accuracy. In this regard, an embodiment of the present invention adopts a deep reinforcement learning method DDQN to learn the optimal attention mechanism embedding strategy, so that when facing different types of target images, the attention mechanism module that is suitable for them can be adaptively selected and embedded in the feature extraction module. Figure 7 As shown in the figure, the image to be detected is input into DDQN, and the output is the attention mechanism embedding strategy. The module that is most suitable for extracting key features from the image to be detected is selected from the five attention mechanism modules and embedded into the feature extraction module to enhance the key feature focusing and extraction capabilities.

[0080] The same DDQN network structure as in the LIAENDRL network is used here, and the principles of the training and optimization process are the same. The main differences are in the action space, state space, and reward function:

[0081] ① Action space. The action space records all the actions that the DDQN agent can take in the environment. The embodiment of the present invention designs 5 attention mechanism embedding modules, each of which has 2 execution actions: processing the module and no operation (not embedding the module). Since the final selected attention mechanism module is one of them or none of them, there are 6 actions in total. The schematic diagram of the action space is as follows Figure 8 shown. Fig. 9 The figure shows the history information of the last four selected actions, which is a 4×6 vector with one-hot encoding, denoted by g, and the initial value is 0. Each time a new vector h0 is constructed for the currently selected action, the dimension of this new vector is 1×6, and the action expressed by the vector is the same as Figure 8 Whenever g is updated, it first scrolls down 1 step in the row dimension, and then assigns g0 to the first row of the current g. When the agent chooses to terminate the action, it means that the current state is the best according to the strategy, and no action is taken, which indicates that the action decision iteration process will end, and the current action vector is output as the final result.

[0082] ② State space. The state represents the information of the current environment and strongly affects the model's decision on the next action. Therefore, the state space should contain enough comprehensive information. In the task of adaptive learning of the optimal attention mechanism embedding strategy, the input image I to be detected, the execution action b obtained from the optimal attention mechanism embedding strategy learning model based on DDQN, and the corresponding reward signal r after the execution of the action.

[0083] ③ Reward function. The reward function is the key to guide the agent to find the best action selection strategy. In the embodiment of the present invention, the agent adjusts the attention mechanism embedding strategy through a series of action sequences, and finally determines the optimal attention mechanism to be embedded in the feature extraction module; the embodiment of the present invention obtains the heat map of the image to be detected from the feature extraction module, and uses it to determine the candidate positioning frame of the target to be detected. It is hoped that during the training process, the feature extraction module embedded in the attention mechanism according to the strategy can effectively adapt to the feature extraction of different types of targets, obtain more accurate key feature information, and thus improve the detection accuracy. At this time, the image to be detected is input into the feature extraction module embedded in the corresponding attention mechanism according to the strategy, and the heat map of the image is obtained. The candidate positioning frame is determined in the range of pixel values ​​greater than σ times the maximum value of the pixel in the heat map (σ∈[0.1,1]), and the IoU between the candidate positioning frame and the calibration frame of the image to be detected is calculated as the reward function to measure the benefit of the agent's action selection. Specifically, the feature extraction module embedded in the corresponding attention mechanism according to the current strategy can increase the IoU between the candidate positioning frame and the calibration frame of the image to be detected, indicating that the strategic action at this time is more conducive to the key feature extraction of the target to be detected, and should be rewarded; on the contrary, it should be punished. By constructing an appropriate objective function, the problem can be transformed into an objective function optimization problem of maximizing the reward, so that the agent can make decisions that meet expectations. The reward function for each action used in the embodiment of the present invention is:

[0084]

[0085] In the formula, Box t+1 Box represents the candidate positioning box of the target to be detected determined from the heat map of the image to be detected obtained by the feature extraction module after the execution of this action. t Box represents the candidate positioning box of the target to be detected determined from the heat map of the image to be detected obtained by the feature extraction module after the execution of this action. pre is the calibration box of the image to be detected, IoU (Box t+1 ,Box pre ) means Box t+1 With Box pre The sign function takes the value of 1 or -1 depending on whether the value of the internal expression is greater than 0. In order to minimize the number of selections, different reward functions are required for the termination action, as follows:

[0086]

[0087] Where s1 represents the initial state, the initial attention mechanism in the feature extraction module is not replaced, Box1 represents the candidate positioning box of the target to be detected determined by the heat map of the image to be detected obtained from the feature extraction module embedded with the default attention mechanism; after this action is executed, IoU (Box t+1 ,Box pre )-IoU(Box1,Box pre ) is greater than T, the reward η is obtained, otherwise the reward is -η.

[0088] ④ Training optimization. The training process of DDQN here is the same as that of the LIAENDRL network. It still introduces the "exploration and exploitation" method and combines the target network and the experience replay pool to improve the training efficiency and model performance. The training data set is a training set consisting of 100 pictures of different types of targets in two scenarios (pictures of small targets in coal mines that are easily occluded, and pictures of steel defects).

[0089] As an implementation method of the embodiment of the present invention, the workflow of the feature extraction module DSAM is as follows:

[0090] In the feature extraction module DSAM proposed in the embodiment of the present invention, the ability to accurately capture the features of complex small targets is enhanced by introducing dynamic snake convolution. At the same time, the channel-priority convolution attention mechanism is embedded to realize the adaptive attention to channel and spatial information in the process of complex small target feature extraction, promote rapid focusing on important information channels and important spatial areas, better suppress the interference of background noise and enhance the expression ability of the extracted features for the target position and shape, thereby improving the target detection network's ability to accurately locate different complex small targets.

[0091] (1) Dynamic snake convolution is introduced to enhance the ability of the feature extraction module to accurately capture the features of complex small targets.

[0092] Complex small targets often show a variety of positions and shapes, resulting in complex and irregular features in these target images. Since most target detection models based on convolutional neural networks (CNNs) use fixed convolution kernels, the receptive field is limited, which limits the model's ability to capture a large amount of local information to a certain extent. Therefore, this traditional fixed convolution structure limits the extraction of complex and irregular features, thereby increasing the difficulty of detecting complex small targets and reducing the generalization ability of the model. In order to enhance the accurate detection capability of the CNN-based target detection model for complex small targets, it is crucial to adaptively expand the receptive field, which will help the detection model learn long-distance and irregular spatial relationships, thereby enhancing the feature extraction performance of the detection model. In this regard, the embodiment of the present invention introduces dynamic snake convolution (DSConv), which learns deformation from the input feature map and adaptively focuses on the tortuous and slender local structure of the target, and expands the local receptive field to more accurately capture the features of the irregular shape structure, thereby improving the detection model's ability to adapt to different target geometries and sizes, thereby promoting the detection model to flexibly identify irregular features in complex small target images.

[0093] The embodiment of the present invention introduces the DSC_Attention module into the feature extraction module to enhance the feature adaptive extraction capability for complex small targets. Figure 2 The figure shows the structure of the DSC_Attention module. The DSC_Attention module is an improved module for the C2f module in the YOLO series of algorithms. A new DCSA module is designed to replace the bottleneck module (Bottleneck) in the original C2f module. The DCSA module includes 2 DSConv modules and 1 attention mechanism module.

[0094] (2) Introducing the attention mechanism to improve the detection model’s ability to accurately locate small targets

[0095] Due to their small scale, irregularity and uneven distribution, the detection of complex small targets is easily affected by background images, external noise, etc., resulting in the reduction of the ability of many existing detection models to accurately locate complex small targets. In recent years, many researchers have proposed various forms of attention mechanisms, such as channel attention, spatial attention, etc., to improve the ability of detection models to focus on key features of targets. However, considering channel attention or spatial attention alone only improves the detection effect of conventional targets to a certain extent, but when facing complex small targets that are susceptible to environmental influences, the detection effect is still not ideal. In this regard, the embodiment of the present invention introduces the channel priority convolutional attention mechanism (Channel Prior Convolutional Attention, CPCA) by default in the DSC_Attention module and the Attention module, so as to realize the dynamic distribution of attention weights in the channel and spatial dimensions while combining the channel attention and spatial attention mechanisms, promote rapid focusing on important information channels and important spatial areas, and improve the ability of the target detection network to accurately locate small targets. In addition, in order to further improve the adaptability of the feature extraction module designed in the embodiment of the present invention when dealing with different types of target feature extraction, an optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning is also designed to achieve adaptive adjustment of the attention mechanism module in the feature extraction module when dealing with different types of targets in other complex environments, so as to achieve more ideal target key feature focusing and extraction. The initial default embedded attention mechanism module of the DSC_Attention module and the Attention module in the feature extraction module is the CPCA attention mechanism module.

[0096] (3) Construct a feature extraction module DSAM based on dynamic snake convolution and attention mechanism

[0097] In order to effectively enhance the ability of the feature extraction network to accurately capture the position and shape features of complex small targets, the embodiment of the present invention designs and constructs a feature extraction module DSAM based on the traditional Conv module, DSConv module, DSC_Attention module, Attention module and SPP module. The structure of the module is as follows: Figure 3As shown in the figure, L1-L11 layers are submodules of the DSAM module, including 2 Conv modules, 3 DSConv modules, 4 DSC_Attention modules, 1 Attention module and 1 SPP module. The initial default embedded attention mechanism module in the DSC_Attention module and the Attention module is the CPCA attention mechanism module. The L3, L5, L7, and L11 layers in this module can output feature maps of different scales as input to the feature fusion network MBiFPN for multi-scale feature fusion.

[0098] As an implementation method of the embodiment of the present invention, the working process of the feature fusion module MBiFPN is as follows:

[0099] In order to make better and more efficient use of the feature information contained in the multi-scale feature maps that is conducive to the classification, recognition and positioning of complex small targets, a new feature fusion module MBiFPN is designed and constructed based on the BiFPN structure, which embeds the designed semantic and detail information fusion module (SDFM), DSCSP module, Upsample module and dynamic snake convolution module. It can achieve efficient fusion of shallow feature maps containing target detail information and deep feature maps containing category semantic information, while improving the positioning sensitivity of complex small targets.

[0100] (1) A semantic and detail information fusion module embedded with coordinate attention and spatial attention mechanisms is introduced to enhance the multi-scale feature fusion performance. In order to better and more efficiently utilize the feature information contained in the multi-scale feature map that is beneficial to target detection and recognition, the SDFM module is designed and introduced into the BiFPN module to fuse feature maps of different scales. The structure of the SDFM module is as follows: Fig.10 The figure shows the internal structure of the SDFM module, taking the fusion of three layers of feature maps of different scales as an example. Assume that the M-layer feature map is in, Represents the feature map of the mth layer. Fig.10 The SDFM module structure diagram shown in the figure is as follows: First, the feature map containing local spatial information and global channel information is obtained through the spatial attention mechanism and coordinate attention mechanism modules. Reuse dynamic snake convolution processing Reduce the number of channels and obtain the feature map Then, each feature map is resized to match the feature map The resolution is expressed as follows:

[0101]

[0102] In the formula, F I ,FD and F U Respectively express Perform identity mapping, adaptive average pooling and bilinear interpolation operations to match its resolution to H m ×W m On this basis, we use 3×3 convolution to Perform smooth convolution operation to obtain The expression is as follows:

[0103]

[0104] Among them, ρ mn Indicates the parameters of the smoothing operation, Represents the nth smoothed feature map at the mth layer. Finally, in order to further enhance the mth layer feature map to integrate more target semantic information and target boundary detail information, the Hadamard product is used to process the mth layer feature map, and the expression is as follows:

[0105]

[0106] Where H(·) represents the Hadamard product operation.

[0107] (2) Dynamic snake convolution is introduced to enhance the target feature extraction capability and improve the sensitivity to complex and irregular small target features. The embodiment of the present invention designs and constructs a DSCSP module based on the traditional Conv module, DSConv module and Concat module. The module can enhance the target feature extraction capability by introducing dynamic snake convolution while reducing the amount of calculation and network complexity. The structure of the DSCSP module is as follows: Fig.11 as shown in .

[0108] (3) Construct the feature extraction module MBiFPN.

[0109] In order to achieve efficient fusion of shallow feature maps containing target detail information and deep feature maps containing category semantic information, and improve the sensitivity to complex small target features, the embodiment of the present invention designs and constructs a new feature fusion module MBiFPN based on the BiFPN structure, the traditional DSConv module, the DSCSP module, the SDFM module and the Upsample module. The structure of this module is as follows: Figure 1As shown in . The bottom-up fusion path in the MBiFPN network includes: 3 SDFM modules are placed in the L13, L16 and L19 layers respectively; 3 DSCSP modules are placed after the 3 SDFM modules (L14, L17 and L20 layers). L12, L15 and L18 are Upsample modules. The top-down path includes 3 groups of identical module structures: L21-L23 layers are constructed by 1 DSConv module, 1 SDFM module and 1 DSCSP module, and L24-L26 layers and L27-L29 layers are also constructed in the same way. After constructing the top-down and bottom-up fusion paths in the feature fusion module respectively through the above modules, based on the design idea of ​​the BiFPN structure, the embodiment of the present invention constructs connections for the feature maps output by the L5 and L7 layers to the L22 and L25 layers respectively.

[0110] The multi-scale feature fusion module MBiFPN constructed in the embodiment of the present invention can realize efficient fusion of multi-scale features while further enhancing the sensitivity of feature extraction of complex small targets, thereby improving the recognition and positioning accuracy of small targets.

[0111] As an implementation method of the present invention, based on the L20, L23, L26, and L29 four-layer output feature maps of the feature fusion module MBiFPN, four DSCSHead target prediction branches are designed and constructed. Each branch is based on the independent decoupled detection head in the YOLOv8 series algorithm, introduces dynamic snake convolution and improves the bounding box regression task loss function, so as to improve the recognition accuracy of complex small targets and accelerate convergence. The structure of the designed DSCSHead module is as follows: Fig.12 As shown. When the original independent decoupled detection head faces the small target detection task that is interfered by complex environmental noise, the slight position deviation of the predicted small pixel target frame will have an adverse effect on the calculation of the bounding box loss function, which can easily lead to a decrease in recognition accuracy; and when faced with small targets with no obvious aspect ratio or proportion changes, the loss function convergence rate is not high, which in turn affects the performance of the entire regression task. To overcome the above problems, an embodiment of the present invention introduces dynamic snake convolution and improves the bounding box regression task loss function, adopts the SIoU loss function to replace the CIoU loss function, and combines DFL to perform the bounding box regression task; the BCE loss function is used for the classification branch to process the target classification task.

[0112] SIoU considers four aspects of cost to construct the target loss function: Angle cost, Distance cost, Shape cost and IoU cost. Fig.13 The parameters related to the SIoU loss function are shown.

[0113] (1) The expression of angle cost is as follows:

[0114] E angle =1-2×sin 2 (arcsin(C h / σ)-(π / 4)) (30)

[0115] In the formula,

[0116]

[0117] (2) The expression of distance cost is as follows:

[0118]

[0119] (3) Shape cost is defined as:

[0120]

[0121] (4) IoU Cost is defined as:

[0122]

[0123] In the formula, B and B GT Represent the true calibration box and the predicted box respectively.

[0124] (5) The definition of SIOU Loss is as follows:

[0125]

[0126] Based on the definition of the SIOU loss function above, the calculation of the distance cost takes into account the angle loss associated with the true marked box and the predicted box. The introduction of the angle cost in the loss function is conducive to calculating the distance cost between the marked box and the predicted box. In the initial stage of detection model training, the predicted box and the marked box usually do not intersect. By introducing the angle cost, the calculation of the distance between the two boxes can be accelerated, thereby achieving faster convergence of the entire regression task. In summary, with the introduction of the angle cost, the SIoU loss function realizes a more comprehensive loss function calculation, while further improving the stability of the loss function convergence, thereby improving the regression accuracy and reducing the prediction error.

[0127] In order to overcome the problems of low detection accuracy and poor model robustness in complex scene small target detection due to uneven scale and irregular visualization features of small targets in complex environments, poor lighting conditions and low image quality, a new detection device for small targets in complex scenes is proposed, which can achieve high-precision recognition and positioning of small targets while having the ability to adapt to multiple complex scenes. The detection device includes four aspects: (1) A complex small target feature extraction module based on dynamic snake convolution and attention mechanism, which effectively enhances the network's ability to extract complex and irregular target features, improves the ability to focus on the detailed features of small targets, and suppresses the interference of complex backgrounds, thereby improving the detection sensitivity of small targets. (2) The image enhancement preprocessing and feature extraction adaptive enhancement module based on reinforcement learning first uses the designed low-quality image adaptive enhancement network based on deep reinforcement learning to adaptively select the optimal image enhancement method combination to enhance the input low-quality image; then, the designed feature extraction module multi-attention mechanism adaptive embedding network based on heat map and DDQN is used to adaptively select the attention mechanism that is more conducive to the extraction of key features for different types of targets and embed it into the feature extraction module, thereby improving the feature extraction module's ability to focus on and extract key features of different types of small targets. (3) The feature fusion module that integrates dynamic snake convolution and multi-attention mechanism realizes the efficient fusion of shallow features containing detail information and deep features containing semantic information, thereby improving the positioning sensitivity of complex small targets; (4) The dynamic snake convolution and SIoU improved detection head are introduced to further improve the positioning accuracy of the target detection model for complex small targets and accelerate the convergence speed of the loss function.

[0128] Embodiment 2:

[0129] The embodiment of the present invention also provides a method for detecting small targets in complex scenes, comprising:

[0130] Adaptive enhancement module is used to perform adaptive enhancement preprocessing on low-quality images containing complex small targets and to extract and enhance key features of different types of targets;

[0131] The feature extraction module adaptively mines a multi-scale feature map representing the key features of small targets from the enhanced image.

[0132] The feature fusion module fuses the shallow feature map containing target detail information and the deep feature map containing category semantic information based on the multi-scale features obtained from the feature extraction module.

[0133] The prediction module performs target classification and positioning based on the fused feature map obtained from the feature fusion module.

[0134] As an implementation method of an embodiment of the present invention, the feature extraction module is constructed based on a DSC_Attention module fused with dynamic snake convolution, a two-dimensional convolution module Conv, a dynamic snake convolution module DSConv, an Attention module, and an SPP module. Among them, the attention mechanism embedding in the DSC_Attention module and the Attention module adopts an optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, and the initial default embedded attention mechanism module is the CPCA attention mechanism module.

[0135] As an implementation method of an embodiment of the present invention, the adaptive enhancement module includes: an image preprocessing module of a low-quality image adaptive enhancement network based on deep reinforcement learning and a feature extraction adaptive enhancement module of a multi-attention mechanism adaptive embedding network based on a heat map and DDQN.

[0136] As an implementation method of an embodiment of the present invention, the feature fusion module includes: 3 DSConv modules, 6 DSCSP modules, 6 SDFM modules and 3 Upsample upsampling modules; wherein, the bottom-up fusion path in the feature fusion module includes: 3 SDFM modules are placed in the L13, L16 and L19 layers respectively; 3 DSCSP modules are placed in the L14, L17 and L20 layers after the 3 SDFM modules respectively; L12, L15 and L18 are Upsample upsampling modules; the top-down path includes 3 groups of identical module structures: L21-L23 layers are constructed by 1 DSConv module, 1 SDFM module and 1 DSCSP module, and L24-L26 layers and L27-L29 layers are also constructed in the same way. After constructing the top-down and bottom-up fusion paths in the feature fusion module respectively through the above modules, based on the design idea of ​​the BiFPN structure, the embodiment of the present invention constructs connections to the L22 and L25 layers for the feature maps output by the L5 and L7 layers respectively.

[0137] Embodiment 3:

[0138] An embodiment of the present invention further provides a complex scene small target detection system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a complex scene small target detection method when executed by the processor.

[0139] Embodiment 4:

[0140] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes the method for detecting small targets in complex scenes when running.

[0141] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A small target detection device in complex scenes, characterized in that: include: Adaptive enhancement module, used to perform adaptive enhancement preprocessing on low-quality images containing complex small targets and to perform feature extraction and enhancement on key features of different types of targets; The feature extraction module is used to adaptively mine multi-scale feature maps that characterize the key features of small targets from the enhanced image; A feature fusion module is used to fuse the shallow feature map containing target detail information and the deep feature map containing category semantic information based on the multi-scale features obtained from the feature extraction module; The prediction module is used to classify and locate the target based on the fused feature map obtained from the feature fusion module.

2. The complex scene small target detection device according to claim 1, characterized in that: The feature extraction module is constructed based on the designed DSC_Attention module, two-dimensional convolution module Conv, dynamic snake convolution module DSConv, Attention module and SPP module. Among them, the embedding of the attention mechanism in the DSC_Attention module and the Attention module adopts the optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, and the initial default embedded attention mechanism module is the CPCA attention mechanism module.

3. The complex scene small target detection device according to claim 2, characterized in that: The adaptive enhancement module includes: a low-quality image adaptive enhancement network based on deep reinforcement learning and a feature extraction adaptive enhancement module based on a heat map and DDQN multi-attention mechanism adaptive embedding network.

4. The complex scene small target detection device according to claim 3, characterized in that: The feature fusion module contains: 3 DSConv modules, 6 DSCSP modules, 6 SDFM modules and 3 Upsample upsampling modules; among them, the bottom-up fusion path in the feature fusion module includes: 3 SDFM modules are placed in the L13, L16 and L19 layers respectively; 3 DSCSP modules are placed in the L14, L17 and L20 layers after the 3 SDFM modules respectively; L12, L15 and L18 are upsampling modules; the top-down path includes 3 groups of identical module structures: L21-L23 layers are constructed by 1 DSConv module, 1 SDFM module and 1 DSCSP module, and L24-L26 layers and L27-L29 layers are also constructed in the same way.

5. A method for detecting small targets in complex scenes, characterized in that: include: Adaptive enhancement module is used to perform adaptive enhancement preprocessing on low-quality images containing complex small targets and to extract and enhance key features of different types of targets; The feature extraction module adaptively mines a multi-scale feature map representing the key features of small targets from the enhanced image. The feature fusion module fuses the shallow feature map containing target detail information and the deep feature map containing category semantic information based on the multi-scale features obtained from the feature extraction module. The prediction module performs target classification and positioning based on the fused feature map obtained from the feature fusion module.

6. The method for detecting small targets in complex scenes as claimed in claim 5, characterized in that: The feature extraction module is constructed based on the designed DSC_Attention module, two-dimensional convolution module Conv, dynamic snake convolution module DSConv, Attention module and SPP module. Among them, the embedding of the attention mechanism module in DSC_Attention and Attention adopts the optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, and the initial default embedded attention mechanism module is the CPCA attention mechanism module.

7. The method for detecting small targets in complex scenes as claimed in claim 6, characterized in that: The adaptive enhancement module includes: a low-quality image adaptive enhancement network based on deep reinforcement learning and a feature extraction adaptive enhancement module based on a heat map and DDQN multi-attention mechanism adaptive embedding network.

8. The method for detecting small targets in complex scenes as claimed in claim 7, characterized in that: The feature fusion module contains: 3 DSConv modules, 6 DSCSP modules, 6 SDFM modules and 3 Upsample upsampling modules; among them, the bottom-up fusion path in the feature fusion module includes: 3 SDFM modules are placed in the L13, L16 and L19 layers respectively; 3 DSCSP modules are placed in the L14, L17 and L20 layers after the 3 SDFM modules respectively; L12, L15 and L18 are Upsample upsampling modules; the top-down path includes 3 groups of identical module structures: L21-L23 layers are constructed by 1 DSConv module, 1 SDFM module and 1 DSCSP module, and L24-L26 layers and L27-L29 layers are also constructed in the same way.

9. A small target detection system in complex scenes, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the method for detecting small targets in complex scenes as described in any one of claims 5 to 8 is executed.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when running, executes the method for detecting small targets in complex scenes as described in any one of claims 5 to 8.

Images