Small target detection method and device and storage medium

Abstract

The invention relates to a small target detection method and device and a storage medium, and the method comprises the steps: obtaining a to-be-processed image; inputting an image to be processed into the trained small target detection model; wherein the small target detection model comprises an enhanced feature extraction network and a detail feature optimization module; performing local feature extraction processing on the to-be-processed image through a plurality of context sensing aggregation units in the enhanced feature extraction network to generate a plurality of local feature maps and deep feature maps with different resolutions, and fusing the plurality of local feature maps to generate a global enhanced feature map; performing detail optimization processing on the input global enhanced feature map and the deep feature map through a detail feature optimization module to generate a prediction layer feature map; and obtaining a target detection result based on the prediction layer feature map. According to the invention, the problem of low detection precision of the small target is solved, the detection capability of the small target is enhanced, and the detection accuracy of the small target is remarkably improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision target detection, and in particular to a small target detection method, device and storage medium. Background Art

[0002] In the field of computer vision, small target detection is a very challenging task, which requires accurate recognition and positioning of small targets in images. These targets are difficult to extract effective information and difficult to detect due to their small size, blurred edges, and susceptibility to background interference. However, small target detection plays a fundamental role in many applications such as automated driving, ocean monitoring, and medical imaging diagnosis. Therefore, how to effectively improve the performance of small target detection has always been a hot topic and difficulty in related research fields.

[0003] In recent years, despite the significant progress made in deep learning and small target detection technology, small target detection still faces many challenges. First, although the multi-scale feature fusion technology can retain shallow fine-grained information and deep semantic information to a certain extent and improve the target detection performance, it still has shortcomings in small target detection. The main reason is that multi-scale feature fusion is usually performed in the feature pyramid stage or in a deeper network, but not in the early backbone network feature extraction stage. Secondly, by constructing and training complex neural networks, deep learning models can automatically extract multi-level and multi-scale features from input data, thereby achieving accurate detection of targets. However, as the number of layers of deep neural networks increases, the receptive field of the network gradually increases, and the resolution of the feature map decreases accordingly, which causes the details of small targets to be easily lost or covered by large target features. This problem seriously restricts the performance of small target detection, resulting in frequent missed detections and false detections.

[0004] Currently, no effective solution has been proposed for the problem of low detection accuracy for small targets in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a small target detection method, device, system, electronic device and storage medium to at least solve the problem of low detection accuracy for small targets in the related art.

[0006] In a first aspect, an embodiment of the present application provides a small target detection method, comprising: Get the image to be processed; Inputting the image to be processed into a trained small target detection model; wherein the small target detection model includes an enhanced feature extraction network and a detail feature optimization module; Performing local feature extraction processing on the image to be processed via multiple context-aware aggregation units in the enhanced feature extraction network to generate multiple local feature maps and deep feature maps with different resolutions, and fusing the multiple local feature maps to generate a global enhanced feature map; The input global enhanced feature map and the deep feature map are optimized and processed in detail via a detail feature optimization module to generate a prediction layer feature map; Based on the prediction layer feature map, a target detection result is obtained.

[0007] In some embodiments, the local feature extraction processing is performed on the image to be processed via multiple context-aware aggregation units in the enhanced feature extraction network to generate multiple local feature maps with different resolutions, including: Performing convolution processing on the image to be processed via a plurality of context-aware aggregation units in the enhanced feature extraction network to obtain a plurality of initial feature maps with different resolutions; A target feature map is determined from each of the initial feature maps, and remaining feature maps in the initial feature maps except the target feature map are upsampled to obtain a plurality of local feature maps of the same scale.

[0008] In some embodiments, the image to be processed is convolved via multiple context-aware aggregation units in the enhanced feature extraction network to obtain multiple initial feature maps of different resolutions, including: Performing a convolution operation on the image via the context-aware aggregation unit to generate a first initial feature map, and performing a depth-separable convolution operation on the first initial feature map to obtain a second initial feature map; the initial feature map includes the first initial feature map and the second initial feature map; The determining of the target feature map from each of the initial feature maps comprises: The first initial feature map in the initial feature maps is determined as the target feature map.

[0009] In some embodiments, fusing the multiple local feature maps to generate a global enhanced feature map includes: Through the context-aware aggregation unit, each of the local feature maps is globally pooled to generate a plurality of pooled local feature maps; and spatially adaptive weight allocation processing is performed on the pooled local feature maps to obtain a first weight parameter corresponding to each of the local feature maps, and based on the first weight parameter, the plurality of local feature maps are fused to generate the global enhanced feature map.

[0010] In some embodiments, the fusing of the plurality of local feature maps based on the first weight parameter to generate the global enhanced feature map includes: Performing weight normalization processing on each of the first weight parameters to obtain a second weight parameter; Based on the second weight parameter, a plurality of the local feature maps are fused to generate the global enhanced feature map.

[0011] In some embodiments, the detail feature optimization module includes a spatial attention mechanism and a channel attention mechanism; the detail feature optimization module optimizes the input global enhanced feature map and the deep feature map to generate a prediction layer feature map, including: The global enhanced feature map is spatially reorganized by the detail feature optimization module to generate a shallow reorganized feature map; wherein the shallow reorganized feature map has the same size as the deep feature map; The shallow reorganized feature map is subjected to the spatial attention mechanism and the channel attention mechanism respectively to extract the spatial weight and the channel weight; based on the spatial weight, the channel weight and the deep feature map, a prediction layer feature map is generated.

[0012] In some embodiments, the step of subjecting the shallow reorganized feature map to the spatial attention mechanism and the channel attention mechanism respectively to extract the spatial weight and the channel weight comprises: Performing convolution operation and global pooling on the shallow reorganized feature map via the spatial attention mechanism to generate the spatial weight; The shallow reorganized feature map is convolved and globally averaged pooled via the channel attention mechanism, and integrated via the fully connected layer in the detail feature optimization module to generate the channel weight.

[0013] In some embodiments, the method further comprises: Get training images; Inputting the training image into the initial feature extraction network for prediction to obtain a training enhancement feature map, and inputting the training enhancement feature map into the initial detail optimization module for prediction to obtain a training layer feature map; Based on the training layer feature map, a loss function is constructed, and the parameters of the detection model are adjusted with the goal of minimizing the calculation result of the loss function to obtain the trained small target detection model.

[0014] In a second aspect, an embodiment of the present application provides a small target detection device, including: An image acquisition module, used for acquiring an image to be processed; A model detection module, used for inputting the image to be processed into a trained small target detection model; wherein the small target detection model includes an enhanced feature extraction network and a detail feature optimization module; The model detection module is also used to perform local feature extraction processing on the image to be processed via multiple context-aware aggregation units in the enhanced feature extraction network, generate multiple local feature maps and deep feature maps with different resolutions, and fuse the multiple local feature maps to generate a global enhanced feature map; The model detection module is also used to optimize the details of the input global enhancement feature map and the deep feature map through the detail feature optimization module to generate a prediction layer feature map; The result generation module is used to obtain the target detection result based on the prediction layer feature map.

[0015] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the small target detection method as described in the first aspect above is implemented.

[0016] Compared with the related art, the small target detection method, device and storage medium provided in the embodiments of the present application balance the detail features of local small targets and the global context by enhancing the feature extraction network, and at the same time use the detail feature optimization module to strengthen the geometry and local detail features in the deep features, so that the network can retain the local and global information of the image at an earlier stage, reduce information loss, and strengthen the geometry and local detail features in the deep features, thereby solving the problem of low detection accuracy of small targets, enhancing the detection capability of small targets, and significantly improving the accuracy of small target detection.

[0017] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a hardware structure block diagram of a terminal of a small target detection method according to an embodiment of the present invention; Figure 2 is a flowchart of a small target detection method according to an embodiment of the present application; Figure 3 is an overall flow chart of a small target detection method according to a preferred embodiment of the present application; Figure 4is a flowchart of an enhanced feature extraction network of a small target detection method according to a preferred embodiment of the present application; Figure 5 is a flowchart of a detail feature optimization module of a small target detection method according to a preferred embodiment of the present application; Figure 6 It is a structural block diagram of a small target detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0020] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0021] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 is a hardware structure block diagram of a terminal of a small target detection method according to an embodiment of the present invention. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0023] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the small target detection method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0024] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0025] This embodiment provides a small target detection method. Figure 2 is a flowchart of a small target detection method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps: Step S201, obtaining an image to be processed; Step S202, inputting the image to be processed into a trained small target detection model; wherein the small target detection model includes an enhanced feature extraction network and a detail feature optimization module; Among them, the image to be processed can come from various image acquisition devices, such as cameras, scanners, etc. The image to be processed contains small target objects, which need to be accurately detected and identified through subsequent steps; small target objects specifically refer to small-sized moving objects that appear in the required scene; the small size can be a size smaller than the preset size value. Usually, the small target object can be a small animal (such as a mouse), small debris (such as falling leaves, paper, plastic bags, etc.) or other objects. In addition, although pedestrians and vehicles are important objects in video structured analysis, when they are far away from the camera and only occupy a small part of the picture, the accuracy of their recognition and analysis may be affected; in this case, according to specific needs, objects such as pedestrians and vehicles in the distance can also be considered as small target objects. The small target detection model includes an enhanced feature extraction network and a detail feature optimization module. The enhanced feature extraction network is a global-local enhanced feature extraction network (GLEFENet), which is composed of multiple stacked context-aware hierarchical feature aggregation modules (CAHFAM). Inputting the image to be processed into the trained small target detection model is the starting point of image processing and provides a basis for subsequent feature extraction and optimization.

[0026] Step S203, performing local feature extraction processing on the image to be processed via multiple context-aware aggregation units in the enhanced feature extraction network, generating multiple local feature maps and deep feature maps with different resolutions, and fusing the multiple local feature maps to generate a global enhanced feature map; Among them, the image to be processed is first processed by multiple context-aware aggregation units (CAHFAM) in the enhanced feature extraction network (GLEFENet). Each context-aware aggregation unit uses a convolution layer to obtain a feature map for the input feature, and then uses depth-separable convolution continuously to obtain feature maps of different resolutions in turn, generating multiple local feature maps of different resolutions. These feature maps have receptive fields of different sizes and context perception capabilities of different scales. Multiple local feature maps are fused to generate a global enhanced feature map, which includes multi-scale local details and global semantic information. The deep feature map is gradually formed in the deep stage of the feature extraction network through continuous convolution, pooling and other operations. In the enhanced feature extraction network (GLEFENet), due to the use of CAHFAM instead of the traditional convolution block, the deep feature map is gradually formed in the process of fusing local details with global semantic information, and retains rich context information. In this step, by enhancing the context-aware aggregation unit (CAHFAM) of the feature extraction network (GLEFENet), multi-scale adaptive feature fusion can be performed in the feature extraction stage, which can retain the local and global information of the image at an earlier stage and balance the local and global information. Due to the fusion of multi-scale local details and global semantic information, GLEFENet can reduce the possibility of small target features being gradually submerged in the network, thereby improving the performance of small target detection.

[0027] Step S204, optimizing the input global enhanced feature map and deep feature map through the detail feature optimization module to generate a prediction layer feature map; Among them, the Fine Characterization Refinement Module (FCRM) uses the rich detail information in the global enhanced feature map (shallow feature map) to supplement the deep feature map to enhance the geometric and local detail features in the deep feature map. The global enhanced feature map and the deep feature map are input into the trained Detail Feature Refinement Module (FCRM) for optimized detail processing. FCRM uses the rich detail information in the global enhanced feature map to supplement the deep features, obtain the geometric features of the target, and enhance the interaction between semantic information and detail information to generate a prediction layer feature map. Since deep feature maps usually contain more abstract and global feature information, but are prone to losing detail information, this step fuses the detail information in the shallow feature map through the FCRM module, effectively enhancing the geometric and local detail features in the deep feature map and improving the model's detection accuracy for small targets.

[0028] Step S205, obtaining the target detection result based on the prediction layer feature map.

[0029] Among them, based on the prediction layer feature map, the existing target detection algorithm (such as bounding box regression, classification, etc.) is used to obtain the final target detection result. The target detection result includes the target's location information (such as center point coordinates, length and width, etc.) and classification information (such as the category to which the target belongs).

[0030] Through the above steps, the present application inputs the image to be processed into the trained small target detection model. The enhanced feature extraction network in the small target detection model replaces the traditional convolution block with the context-aware aggregation unit (CAHFAM), and performs multi-scale adaptive feature fusion in the early stage of feature extraction. In the feature extraction layer of the model, the image to be processed is processed into multiple local feature maps of different resolutions, and then these multiple local feature maps are fused to achieve feature enhancement effect, so that the generated global feature enhancement map not only contains the detailed information of the local features, but also incorporates the global context information to generate a global enhanced feature map. Therefore, the global enhanced feature map pays attention to both local information and global information of the image, achieving a balance between local and global information; the deep feature map is gradually formed in the process of fusing local details with global semantic information. The deep feature map has a larger receptive field and can capture more abstract and global feature information, but it is often easy to lose the detailed information of small targets. Therefore, in order to make up for this defect, the global enhanced feature map and the deep feature map are combined. Figure 1 The model uses the rich detail information in the global enhanced feature map to supplement the deep features, strengthen the geometric and local detail features in the deep features, and enhance the interaction between semantic information and detail information. Finally, the prediction layer feature map is generated. The prediction layer feature map contains more complete and accurate target feature information. Finally, the target detection result is output based on the prediction layer feature map. This application solves the problem of low detection accuracy of small targets in existing small target detection methods due to the difficulty of feature extraction and easy loss of detail information by jointly using the enhanced feature extraction network and the detail feature optimization module, effectively improving the performance of the model in small target detection and making the detection results more accurate and reliable.

[0031] In some of the embodiments, a plurality of context-aware aggregation units in an enhanced feature extraction network are used to perform local feature extraction processing on the image to be processed to generate a plurality of local feature maps of different resolutions, including: The image to be processed is convolved through multiple context-aware aggregation units in the enhanced feature extraction network to obtain multiple initial feature maps with different resolutions; A target feature map is determined from each initial feature map, and the remaining feature maps except the target feature map in the initial feature map are upsampled to obtain multiple local feature maps of the same scale.

[0032] In the enhanced feature extraction network (global-local enhanced feature extraction network GLEFENet), the context-aware aggregation unit (CAHFAM) is used instead of the traditional convolution block. The input data is extracted from the processed image through multiple convolution layers to obtain multiple initial feature maps with different resolutions. One of the multiple initial feature maps is selected as the target feature map. Generally, the feature map obtained after the first convolution process is determined as the target feature map. Then, the scale of the remaining feature maps except the target feature map is adjusted to the same as the target feature map through upsampling. Figure 1 To facilitate subsequent feature fusion, specifically, upsampling can be achieved through interpolation methods (such as bilinear interpolation, nearest neighbor interpolation, etc.) or other neural network layers (such as transposed convolutional layers). This embodiment uses upsampling to make local feature maps of different resolutions have the same scale, which facilitates subsequent feature fusion and ensures the consistency between feature maps during the fusion process. By fusing feature information of different scales, the network can learn richer and more diverse feature representations, which helps to improve the accuracy and robustness of small target detection.

[0033] In some embodiments, a plurality of context-aware aggregation units in an enhanced feature extraction network are used to perform convolution processing on the image to be processed to obtain a plurality of initial feature maps of different resolutions, including: Performing a convolution operation on the image through a context-aware aggregation unit to generate a first initial feature map, and performing a depth-separable convolution operation on the first initial feature map to obtain a second initial feature map; the initial feature map includes the first initial feature map and the second initial feature map; Determine the target feature map from each initial feature map, including: A first initial feature map among the initial feature maps is determined as a target feature map.

[0034] Among them, a standard convolution operation is performed on the image to be processed. This step usually uses multiple convolution kernels (or filters) to slide on the image, perform weighted summation on each local area, and may add a bias term, and then pass a nonlinear activation function (such as ReLU) to obtain the first initial feature map. It should be noted that the first initial feature map obtained after the first convolution process is used as the target feature map in order to ensure that the feature map sizes output by adjacent multi-scale adaptive feature fusion processes do not differ too much, otherwise too much information will be lost, and it is difficult to maintain the integrity of the target details and context information contained in the feature map.

[0035] Then, a depthwise separable convolution operation can be performed on the first initial feature map to generate a second initial feature map. The depthwise separable convolution operation includes two steps: channel-by-channel convolution (Depthwise Convolution) and pointwise convolution (Pointwise Convolution). Channel-by-channel convolution uses a different convolution kernel for each input channel and outputs a feature map with the same number of input channels; pointwise convolution uses a 1x1 convolution kernel to perform a linear combination of the output of channel-by-channel convolution between channels to obtain a new feature map. By adjusting the number of convolution kernels of the pointwise convolution, the number of output feature maps can be controlled. Based on the first initial feature map (target feature map), a depthwise separable convolution operation can be performed to generate multiple local feature maps of different resolutions, namely the second initial feature map. These feature maps are related to the target feature map. Figure 1 This embodiment can effectively extract multi-level and multi-scale feature information from the original image through convolution operations, and can generate multiple local feature maps of different resolutions through depth-separable convolution operations. These feature maps contain rich local information and contextual information, which helps to improve the accuracy of small target detection.

[0036] In some embodiments, multiple local feature maps are fused to generate a global enhanced feature map, including: Through the context-aware aggregation unit, each local feature map is globally pooled to generate multiple pooled local feature maps; and spatial adaptive weight allocation processing is performed on the pooled local feature maps to obtain the first weight parameter corresponding to each local feature map, and based on the first weight parameter, the multiple local feature maps are fused to generate a global enhanced feature map.

[0037] Among them, for each local feature map, global average pooling (GAP) and global maximum pooling (GMP) operations are performed respectively. The global pooling operation can extract the global information of the feature map, that is, the average or maximum value operation is performed on the entire feature map to obtain an output vector of fixed length. Through global pooling, each local feature map is converted into a pooled local feature map with global context information. For each pooled local feature map, a convolution layer and an activation function (such as a Sigmoid function) are used to perform adaptive weight adjustment of the spatial position. This process assigns an adaptive weight to each spatial position, that is, the first weight parameter, which reflects the importance of the position in the feature fusion process. Each local feature map is weighted and fused with its corresponding first weight parameter. Specifically, each local feature map is element-wise multiplied with its spatial adaptive weight to emphasize or suppress the features of a specific position. The weighted local feature maps are fused to generate a global enhanced feature map, so that the global enhanced feature map combines information from feature maps of different scales to form a global feature representation containing richer context information. During the fusion process, the weight parameter of each enhanced feature determines its contribution to the final fused feature map. Through the fusion process, the network can generate a global enhanced feature map containing features of multiple scales and multiple positions. This feature map not only retains the global context information, but also highlights the key local details. The adaptive weight adjustment process is shown in the following formula: ; In the above formula, (i=0,1,2) represents the local feature map F i The feature map after resizing; GAP and GMP represent global average pooling and global maximum pooling operations respectively; CAT is the feature concatenation operation; Conv represents the convolution operation; σ is the Sigmoid activation function; W i (i=0,1,2) represents the local feature map F i The first weight parameter of .

[0038] In this embodiment, through the global pooling operation, each local feature map is converted into a feature representation with global context information, which helps the network to better understand and utilize the global structural information of the image during the feature fusion process; the spatial adaptive weight allocation processing assigns different weights to different spatial positions of each local feature map, so that key information can be more accurately retained and emphasized during the feature fusion process, while irrelevant information can be suppressed; the global enhanced feature map fuses information from feature maps of different scales and emphasizes the feature representation of key areas, so that the network can better identify and locate small targets in the image, thereby improving the performance of small target detection.

[0039] In some embodiments, based on the first weight parameter, multiple local feature maps are fused to generate a global enhanced feature map, including: Performing weight normalization processing on each first weight parameter to obtain a second weight parameter; Based on the second weight parameter, multiple local feature maps are fused to generate a global enhanced feature map.

[0040] Among them, after determining each local feature map and its first weight parameter, these first weight parameters are normalized to obtain the second weight parameter. Normalization usually uses a Softmax function or a similar mechanism to map the weight parameter to the [0,1] interval and ensure that the sum of all weight parameters is 1. The weight normalization process in this embodiment is to ensure that during the feature fusion process, the contribution of each global enhancement feature map can be balanced according to its importance, avoiding the loss or over-amplification of feature information due to excessive or too small weights.

[0041] The specific process is shown in the following formula: ; ; In the above formula, H represents the normalized second weight parameter; Softmax refers to the normalized exponential function; CAT is the feature concatenation operation; W i (i=0,1,2) represents the local feature map F i The first weight parameter of (i=0,1,2) represents the local feature map F i Feature map after resizing; F out It is the global enhanced feature map output after the enhanced feature extraction network.

[0042] In some embodiments, the detail feature optimization module includes a spatial attention mechanism and a channel attention mechanism; the detail feature optimization module optimizes the details of the input global enhanced feature map and the deep feature map to generate a prediction layer feature map, including: Through the detail feature optimization module, the global enhanced feature map is spatially reorganized to generate a shallow reorganized feature map; wherein the shallow reorganized feature map has the same size as the deep feature map; The shallow reorganized feature map is subjected to the spatial attention mechanism and the channel attention mechanism respectively to extract the spatial weight and channel weight; based on the spatial weight, channel weight and deep feature map, the prediction layer feature map is generated.

[0043] Among them, the detail feature optimization module (FCRM) uses a reorganization operation (i.e., Split operation) to replace the traditional convolution or pooling operation to retain more shallow feature information. The Split operation rearranges the spatial elements of the feature into the channel dimension, thereby adjusting the feature size to match the deep feature. Figure 1 While maintaining consistency, it can preserve spatial encoding information, and thus generate a shallow reorganized feature map. The spatial attention mechanism is to pass the shallow reorganized feature map through the convolution layer and global pooling operation to generate spatial weights. This process focuses on the spatial position information of the feature map, and emphasizes or suppresses the features of specific areas by assigning different weights to different positions. The channel attention mechanism is to pass the shallow reorganized feature map through the convolution layer and global average pooling operation, and integrate it through the fully connected layer to generate channel weights. This process focuses on the channel information of the feature map, and emphasizes or suppresses the features of specific channels by assigning different weights to different channels. Then, the deep feature map is reweighted using the extracted spatial weights and channel weights. Specifically, the spatial weights and channel weights are respectively multiplied or added element by element with the deep feature map to highlight key information and suppress irrelevant information. The reweighted deep feature map is combined with the global enhancement feature map (or the appropriately processed shallow reorganized feature map) to form the final prediction layer feature map. The process is shown in the following formula: ; In the above formula, Refers to the prediction layer feature map output after being enhanced by the detail feature optimization module; represents the deep feature map; T S Represents the function of obtaining spatial weight; T C Represents the function of obtaining channel weights; Represents the shallow feature map.

[0044] This embodiment reorganizes and extracts bidirectional weights on the global enhanced feature map through the detail feature optimization module (FCRM), which can effectively supplement the small target detail information lost in the deep feature map, enhance the interaction between semantic information and detail information, thereby improving the key feature extraction capability of small targets and significantly improving the accuracy of small target detection. The detail feature optimization module retains more shallow feature information and re-weights the deep feature map, which enables the model to more accurately capture the key features of small targets of different scales and complex backgrounds.

[0045] In some embodiments, the shallow reorganized feature map is subjected to a spatial attention mechanism and a channel attention mechanism respectively to extract the spatial weight and the channel weight, including: The shallow reorganized feature map is convolved and globally pooled through the spatial attention mechanism to generate spatial weights; The shallow reorganized feature map is convolved and globally averaged pooled via the channel attention mechanism, and integrated via the fully connected layer in the detail feature optimization module to generate channel weights.

[0046] Among them, after obtaining the shallow reorganized feature map, the spatial weight is generated through the spatial attention mechanism, and the shallow reorganized feature map is convolved to extract the spatial position information of the feature map. The convolution operation can retain the local structural information of the feature map while reducing the amount of calculation; then the convolved feature map is subjected to a global pooling operation (such as global average pooling or global maximum pooling) to obtain the global context information of the feature map. Global pooling compresses the spatial dimension of the feature map into a vector of a fixed length so that the features at each position can contribute to the entire feature map; finally, the feature vector after global pooling is processed through a convolution layer and an activation function (such as a Sigmoid function) to generate a spatial weight. The spatial weight reflects the importance of different positions in the feature map, and emphasizes or suppresses the features of a specific area by assigning different weights to different positions. The channel weights are generated through the channel attention mechanism. Similar to the spatial attention mechanism, the shallow reorganized feature map is first convolved to extract the channel information of the feature map; then the feature map after convolution is subjected to global average pooling to obtain the channel-level global context information of the feature map. Global average pooling compresses each channel of the feature map into a scalar value, reflecting the importance of the channel in the entire feature map; the feature vector after global average pooling is input into the fully connected layer for integration to generate channel weights. The fully connected layer can perform nonlinear transformation on the feature vector and enhance the discriminability of the feature representation through activation functions (such as ReLU or Sigmoid function). The channel weights reflect the importance of different channels in the feature map, and emphasize or suppress the features of specific channels by assigning different weights to different channels. In short, the spatial weights can be adaptively adjusted through the convolution layer, and global pooling can be performed along the channel (it can be global average pooling or global maximum pooling. Global maximum pooling is usually used to capture the most significant features, while global average pooling is used to capture overall features). Finally, the spatial weights are obtained. The spatial weights reflect the importance of different spatial positions and are used to weight the feature map later. At the same time, another convolutional layer is used to adaptively adjust the channel weights, and then global average pooling is used to aggregate spatial information of each channel and obtain global information. Finally, after two fully connected layers, the channel weights are adaptively adjusted again. The channel weights reflect the importance of different channels and are also used to weight the feature maps later. The process is shown in the following formula: ; ; Among them, CAP means global pooling along the channel; GAP refers to the global average pooling operation; fc refers to the fully connected layer; δ represents the ReLu activation function; σ represents the Sigmoid activation function.

[0047] This embodiment generates spatial weights so that the model can focus on the importance of different spatial positions in the image, suppress irrelevant areas while highlighting key areas, which is particularly important for small target detection, because small targets usually only occupy a small part of the image, and spatial weights can help the model focus more on the locations of these small targets; and through the generation of channel weights, the model can distinguish the importance of different channels, so as to more effectively utilize the information in the feature map. For the small target detection task, some channels may contain more key information about the small targets, and channel weights can help the model capture this information more accurately; combined with spatial weights and channel weights, the detail feature optimization module (FCRM) can strengthen the geometric and local detail features in the deep features, enhance the interaction between semantic information and detail information, and improve the key feature extraction capability of small targets, thereby significantly improving the accuracy of small target detection. At the same time, by adaptively adjusting the feature weights of different spaces and channels, the model can more robustly cope with complex and changing scenes.

[0048] In some embodiments, the method further comprises: Get training images; The training image is input into the initial feature extraction network for prediction to obtain a training enhanced feature map, and the training enhanced feature map is input into the initial detail optimization module for prediction to obtain a training layer feature map; Based on the feature map of the training layer, a loss function is constructed, and the parameters of the detection model are adjusted with the goal of minimizing the calculation result of the loss function to obtain a trained small target detection model.

[0049] Among them, images containing small targets are selected from the training data set as training images. These images should cover different scenes, lighting conditions and target postures to ensure the generalization ability of the model; the training images first pass through the initial feature extraction model, which uses the context-aware aggregation unit (CAHFAM) to perform multi-scale feature fusion and extract the training enhanced feature map containing local and global information; the training enhanced feature map is then input into the initial detail optimization model, which uses the rich detail information in the training enhanced feature map (shallow feature) to supplement the deep feature, and generates the enhanced training layer feature map through the steps of Split operation, channel and spatial bidirectional weight extraction, etc.; using the training layer feature The feature map predicts target classification and position regression, and calculates the loss function value based on the difference between the predicted result and the true label. The loss function can include multiple parts such as classification loss, confidence loss and regression loss. Through the back-propagation algorithm, the gradient of the loss function result is passed back to the initial feature extraction model and the initial detail optimization model. According to the gradient information, the weight parameters of the model are updated to minimize the loss function value. This process is carried out through multiple iterations until the model converges or reaches the preset number of training rounds. After iterative training, it gradually converges to form the final enhanced feature extraction network (GLEFENet) and detail feature optimization module (FCRM), that is, the trained small target detection model is obtained.

[0050] Specifically, the loss function includes three categories: classification loss, confidence loss and bounding box regression loss. The formula is as follows: ; ; ; ; In the above formula, , and They are classification loss, confidence loss and regression loss respectively; classes indicates the target type; is the true / predicted confidence (score); Represents the (horizontal center position, vertical center position, width, height) of the predicted / true box; Refers to cross entropy loss; H / W represents the height / width of the final prediction image; M represents the number of anchor boxes; Indicates whether there is an object in the i-th cell and whether the j-th bounding box predictor in the i-th cell is responsible for the prediction.

[0051] By enhancing the joint training of the feature extraction network and the detail feature optimization module, the model in this embodiment can more effectively extract and utilize the local detail features and global context information of small targets, thereby improving the accuracy and robustness of small target detection. Through the iterative training process of the loss function, the structure of the model (including weight parameters) is continuously optimized, making the model more efficient and accurate in extracting features, performing classification and position regression, etc.

[0052] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0053] Figure 3 is an overall flow chart of a small target detection method according to a preferred embodiment of the present application, such as Figure 3 As shown in the figure, the enhanced feature extraction network (i.e., global-local enhanced feature extraction network GLEFENet) is integrated into the feature extraction network by using context-aware aggregation units instead of convolution blocks. Multi-scale feature adaptive fusion is performed in the early stage of the network, that is, the feature extraction stage, and progressive feature fusion is continuously performed in this stage to achieve a balance between local details and global semantics, reducing the possibility of small target features being gradually submerged in the network. In a convolutional neural network, as the network layer becomes deeper, its receptive field will also become larger, and small target feature information will often be submerged by large target feature information. How to retain small target features so that they are not lost is crucial for small target detection models.

[0054] In the enhanced feature extraction network of this application, a context-aware aggregation unit CAHFAM is proposed. This module interactively fuses multi-level feature maps, so that the network can not only focus on the detailed local information such as the edge and texture of small targets, but also have rich global context information.

[0055] Figure 4 is a flowchart of an enhanced feature extraction network of a small target detection method according to a preferred embodiment of the present application, such as Figure 4 As shown in the figure, first use the convolution layer to obtain the feature map F0 for the input feature F, then use the depth-separable convolution continuously to obtain the feature maps F1 and F2 in turn. At this time, three layers of feature maps F with different resolutions are obtained. 0、 F1, F2, these three feature maps have different sizes of receptive fields and different scales of context perception. Then, the upsampling layer is used to resize F1 and F2 to make them consistent with the size of the F0 feature map, and finally the adjusted feature map is obtained. Then, the resized features Perform global average pooling and global maximum pooling respectively to obtain sufficient global information, then perform convolution and activation functions to perform adaptive weight adjustment of spatial positions, and assign an adaptive weight to each spatial position. The process is shown in the following formula: ; In the above formula, (i=0,1,2) represents the local feature map F i The feature map after resizing; GAP and GMP represent global average pooling and global maximum pooling operations respectively; CAT is the feature concatenation operation; Conv represents the convolution operation; σ is the Sigmoid activation function; W i (i=0,1,2) represents the local feature map F i The first weight parameter of .

[0056] At this time, for W i (i=0,1,2) for splicing, normalize the spatial position weight, obtain the normalized spatial position weight, i.e., the second weight parameter, and re-weight the feature map with the second weight parameter, so that the network can achieve the perception of different spatial positions in the image, suppress irrelevant areas and highlight key areas. The process is shown in the following formula: ; ; In the above formula, H represents the normalized second weight parameter; Softmax refers to the normalized exponential function; CAT is the feature concatenation operation; W i (i=0,1,2) represents the local feature map F i The first weight parameter of (i=0,1,2) represents the local feature map F i Feature map after resizing; F out It is the shallow feature map output after the enhanced feature extraction network.

[0057] The detail feature optimization module FCRM proposed in this application considers that deep features have a larger receptive field, focuses more on generating abstract and global features for the target, and is more likely to lose the detail information of the target. Losing detail information is disastrous for small targets, which means that the features of small targets are more likely to be lost in the deep feature layer than in the shallow feature layer. The detail feature optimization module uses shallow features containing rich detail information to supplement the local information of deep features, obtain the geometric features of the target, enhance the interaction between semantic information and detail information, and strengthen the key features of small targets. Figure 5 is a flowchart of a detail feature optimization module of a small target detection method according to a preferred embodiment of the present application, such as Figure 5 As shown in the figure, in order to retain as much shallow feature information as possible, this module first uses the Split operation to replace the convolution or pooling operation, that is, rearranging the spatial elements of the feature into the channel dimension, and then achieving the same as the deep feature The purpose of size consistency is to preserve spatial encoding information, and the feature .then, The features are bidirectionally weighted for channel and space, the deep features are re-weighted, and the key information is bidirectionally highlighted, and finally the deep features that supplement the shallow feature information are obtained. The process is shown in the following equation: ; In the above formula, Refers to the prediction layer feature map output after being enhanced by the detail feature optimization module; represents the deep feature map; T S Represents the function of obtaining spatial weight; T C Represents the function of obtaining channel weights; Represents the shallow feature map.

[0058] Spatial weights and channel weights are generated by the following operations. First, the spatial weights are adaptively adjusted through the convolution layer, and then global pooling is performed along the channel to finally obtain the spatial adaptive weights. At the same time, the channel weights are adaptively adjusted through another convolution layer, and then the spatial information of each channel is aggregated through global average pooling to obtain global information, and finally the channel weights are adaptively adjusted again through two fully connected layers. The process is shown in the following formula: ; ; Among them, CAP means global pooling along the channel; GAP refers to the global average pooling operation; fc refers to the fully connected layer; δ represents the ReLu activation function; σ represents the Sigmoid activation function.

[0059] Next is the specific process of iterating through the loss function during model training: Object detection mainly includes two purposes: classification and positioning. Therefore, the loss function of the model proposed in this application includes three types: classification loss, confidence loss, and bounding box regression loss. The formula is as follows: ; ; ; ; In the above formula, , and They are classification loss, confidence loss and regression loss respectively; classes indicates the target type; is the true / predicted confidence (score); Represents the (horizontal center position, vertical center position, width, height) of the predicted / true box; Refers to cross entropy loss; H / W represents the height / width of the final prediction image; M represents the number of anchor boxes; Indicates whether there is an object in the i-th cell and whether the j-th bounding box predictor in the i-th cell is responsible for the prediction.

[0060] In summary, firstly, the enhanced feature extraction network (i.e., global-local enhanced feature extraction network, GLEFENet) proposed in this application replaces the traditional convolutional block with the context-aware aggregation unit (CAHFAM), performs multi-scale adaptive feature fusion in the feature extraction stage, and continuously fuses local details and global semantic information. The context-aware aggregation unit balances the detail features of local small targets with the global context, reduces the risk of small target features being overwhelmed by large target features in the deep network, and thus improves the small target detection performance. Secondly, the detail feature optimization module (FCRM) proposed in this application supplements the deep features with the rich detail information in the shallow features to solve the problem that the deep features are prone to lose the detail information of small targets. This module strengthens the geometric and local detail features in the deep features, enhances the interaction between semantic information and detail information, thereby improving the key feature extraction capability of small targets and significantly improving the accuracy of small target detection. Finally, the small target detection method proposed in this application combines the enhanced feature extraction network and the detail feature optimization module, which effectively improves the performance of the model in small target detection.

[0061] This embodiment also provides a small target detection device, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0062] Figure 6 is a structural block diagram of a small target detection device according to an embodiment of the present application, such as Figure 6 As shown, the device comprises: An image acquisition module 10 is used to acquire an image to be processed; The model detection module 20 is used to input the image to be processed into the trained small target detection model; wherein the small target detection model includes an enhanced feature extraction network and a detail feature optimization module; The model detection module 20 is also used to perform local feature extraction processing on the image to be processed via multiple context-aware aggregation units in the enhanced feature extraction network, generate multiple local feature maps and deep feature maps with different resolutions, and fuse the multiple local feature maps to generate a global enhanced feature map; The model detection module 20 is also used to optimize the details of the input global enhancement feature map and the deep feature map through the detail feature optimization module to generate a prediction layer feature map; The result generation module 30 is used to obtain the target detection result based on the prediction layer feature map.

[0063] The above-mentioned small target detection device also includes a training module; the training module is used to obtain a training image; the training image is input into the initial feature extraction network for prediction to obtain a training enhanced feature map, and the training enhanced feature map is input into the initial detail optimization module for prediction to obtain a training layer feature map; based on the training layer feature map, a loss function is constructed, and the parameters of the detection model are adjusted with the goal of minimizing the calculation result of the loss function to obtain a trained small target detection model.

[0064] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0065] In addition, in combination with the small target detection method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any small target detection method in the above embodiment is implemented.

[0066] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A small target detection method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a trained small target detection model; wherein the small target detection model includes an enhanced feature extraction network and a detail feature optimization module; Performing local feature extraction processing on the image to be processed via multiple context-aware aggregation units in the enhanced feature extraction network to generate multiple local feature maps and deep feature maps with different resolutions, and fusing the multiple local feature maps to generate a global enhanced feature map; The input global enhanced feature map and the deep feature map are optimized and processed in detail via a detail feature optimization module to generate a prediction layer feature map; Based on the prediction layer feature map, a target detection result is obtained.

2. The small target detection method according to claim 1, characterized in that: The method further comprises: performing local feature extraction processing on the image to be processed via multiple context-aware aggregation units in the enhanced feature extraction network to generate multiple local feature maps with different resolutions, including: Performing convolution processing on the image to be processed via a plurality of context-aware aggregation units in the enhanced feature extraction network to obtain a plurality of initial feature maps with different resolutions; A target feature map is determined from each of the initial feature maps, and remaining feature maps in the initial feature maps except the target feature map are upsampled to obtain a plurality of local feature maps of the same scale.

3. The small target detection method according to claim 2, characterized in that: The convolution processing is performed on the image to be processed via multiple context-aware aggregation units in the enhanced feature extraction network to obtain multiple initial feature maps with different resolutions, including: Performing a convolution operation on the image via the context-aware aggregation unit to generate a first initial feature map, and performing a depth-separable convolution operation on the first initial feature map to obtain a second initial feature map; the initial feature map includes the first initial feature map and the second initial feature map; The determining of the target feature map from each of the initial feature maps comprises: The first initial feature map in the initial feature maps is determined as the target feature map.

4. The small target detection method according to claim 1, characterized in that: The fusing of the plurality of local feature maps to generate a global enhanced feature map comprises: Through the context-aware aggregation unit, each of the local feature maps is globally pooled to generate a plurality of pooled local feature maps; and spatially adaptive weight allocation processing is performed on the pooled local feature maps to obtain a first weight parameter corresponding to each of the local feature maps, and based on the first weight parameter, the plurality of local feature maps are fused to generate the global enhanced feature map.

5. The small target detection method according to claim 4, characterized in that: The fusing the plurality of local feature maps based on the first weight parameter to generate the global enhanced feature map comprises: Performing weight normalization processing on each of the first weight parameters to obtain a second weight parameter; Based on the second weight parameter, a plurality of the local feature maps are fused to generate the global enhanced feature map.

6. The small target detection method according to claim 1, characterized in that: The detail feature optimization module includes a spatial attention mechanism and a channel attention mechanism; the detail feature optimization module optimizes the input global enhanced feature map and the deep feature map to generate a prediction layer feature map, including: The global enhanced feature map is spatially reorganized by the detail feature optimization module to generate a shallow reorganized feature map; wherein the shallow reorganized feature map has the same size as the deep feature map; The shallow reorganized feature map is subjected to the spatial attention mechanism and the channel attention mechanism respectively to extract the spatial weight and the channel weight; based on the spatial weight, the channel weight and the deep feature map, a prediction layer feature map is generated.

7. The small target detection method according to claim 6, characterized in that: The step of subjecting the shallow reorganized feature map to the spatial attention mechanism and the channel attention mechanism respectively to extract the spatial weight and the channel weight comprises: Performing convolution operation and global pooling on the shallow reorganized feature map via the spatial attention mechanism to generate the spatial weight; The shallow reorganized feature map is convolved and globally averaged pooled via the channel attention mechanism, and integrated via the fully connected layer in the detail feature optimization module to generate the channel weight.

8. The small target detection method according to claim 1, characterized in that: The method further comprises: Get training images; Inputting the training image into the initial feature extraction network for prediction to obtain a training enhancement feature map, and inputting the training enhancement feature map into the initial detail optimization module for prediction to obtain a training layer feature map; Based on the training layer feature map, a loss function is constructed, and the parameters of the detection model are adjusted with the goal of minimizing the calculation result of the loss function to obtain the trained small target detection model.

9. A small target detection device, characterized in that: include: An image acquisition module, used for acquiring an image to be processed; A model detection module, used for inputting the image to be processed into a trained small target detection model; wherein the small target detection model includes an enhanced feature extraction network and a detail feature optimization module; The model detection module is also used to perform local feature extraction processing on the image to be processed via multiple context-aware aggregation units in the enhanced feature extraction network, generate multiple local feature maps and deep feature maps with different resolutions, and fuse the multiple local feature maps to generate a global enhanced feature map; The model detection module is also used to optimize the details of the input global enhancement feature map and the deep feature map through the detail feature optimization module to generate a prediction layer feature map; The result generation module is used to obtain the target detection result based on the prediction layer feature map.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the small target detection method according to any one of claims 1 to 8 when running.

Images