A dual-mode real-time small target detection method, device, medium and equipment

Abstract

The present invention belongs to the field of computer vision application technology, and specifically discloses a dual-modal real-time small target detection method, device, medium and equipment. The dual-modal real-time small target detection method includes: obtaining a color image to be detected; inputting the color image to be detected into an image segmentation network model to obtain a segmentation mark image; inputting the color image to be detected and the segmentation mark image into a pre-trained target detection and recognition network to obtain category information and anchor frame information of the target to be detected; wherein the pre-trained target detection and recognition network is composed of a dual-channel image fusion component, a re-parameterized enhanced feature extraction component, a multi-scale feature fusion component and a target prediction component. The present invention improves the detection accuracy, and the fusion of dual-modal information can capture more features of small targets and reduce missed detections. At the same time, it can meet the real-time requirements and can work stably in complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision applications, and specifically relates to a dual-modal real-time small target detection method, device, medium and equipment. Background Art

[0002] In modern science and technology and industry, target detection technology plays a vital role.

[0003] With the continuous expansion and diversification of application scenarios, the demand for small target detection is growing. In many fields such as military reconnaissance, security monitoring, medical image analysis, and traffic supervision, small target detection is often a key link.

[0004] For example, in the military, small aircraft or hidden military facilities at long distances may be important reconnaissance targets; in security monitoring, accurate detection of small targets such as tiny suspicious objects or unknown persons at a distance is of decisive significance for security prevention. However, small targets face many challenges in the detection process due to their small size and unclear features.

[0005] Traditional target detection methods often suffer from poor detection accuracy and high missed detection rate when dealing with small targets due to low resolution and limited information, making it difficult to meet the dual requirements of accuracy and real-time small target detection in practical applications.

[0006] At the same time, in the current development of detection technology, single-modal detection methods have gradually become difficult to adapt to complex and changing environments. Different modal information has its own advantages and limitations. Currently, there is an urgent need for a method that can comprehensively utilize the advantages of different modalities to improve the performance of small target detection.

[0007] Therefore, developing a dual-modal real-time small target detection method has great practical significance and application value, which can effectively overcome the shortcomings of existing technologies and meet the needs of different fields for fast and accurate detection of small targets.

[0008] Based on the above problems, the present invention provides a dual-modal real-time small target detection method, device, medium and equipment. Summary of the invention

[0009] Purpose of the invention: The purpose of the present invention is to provide a dual-modal real-time small target detection method, device, medium and equipment.

[0010] Technical solution: To achieve the above purpose, the present invention is implemented by adopting the following technical solution:

[0011] In a first aspect, the present invention provides a dual-modal real-time small target detection method, comprising:

[0012] Acquire a color image to be detected;

[0013] Inputting the color image into an image segmentation network model to obtain a segmentation label image;

[0014] Inputting the color image to be detected and the segmented marked image into a pre-trained target detection and recognition network to obtain category information and anchor frame information of the target to be detected;

[0015] The pre-trained target detection and recognition network includes: a dual-channel image fusion component, a re-parameterized enhanced feature extraction component, a multi-scale feature fusion component and a target prediction component.

[0016] In combination with the first aspect, optionally, the image segmentation network model includes: an encoder component, a self-attention and multi-layer perception component, a long-distance skip link component and a decoder component.

[0017] In combination with the first aspect, optionally, the dual-channel image fusion component is used for image fusion, and the steps include:

[0018] Normalizing the color image to obtain normalized three-channel information of the color image;

[0019] The normalized three-channel information of the color image and the segmentation mark image information are pixel-level spliced ​​to obtain fused image information, wherein the fused image information includes four channels, namely, an R channel, a G channel, and a B channel of the color image and a mark value channel of the segmentation mark image.

[0020] In combination with the first aspect, optionally, the re-parameterized enhanced feature extraction component includes: a first re-parameterized enhancement module, a second re-parameterized enhancement module, a third re-parameterized enhancement module, and a fourth re-parameterized enhancement module, wherein the output of the first re-parameterized enhancement module serves as the input of the second re-parameterized enhancement module, the output of the second re-parameterized enhancement module serves as the input of the third re-parameterized enhancement module, and the output of the third re-parameterized enhancement module serves as the input of the fourth re-parameterized enhancement module.

[0021] In combination with the first aspect, optionally, the image downsampling rate of the first parameterized enhancement module is 1, the image downsampling rate of the second parameterized enhancement module is 1 / 8, the image downsampling rate of the third parameterized enhancement module is 1 / 16, and the image downsampling rate of the fourth parameterized enhancement module is 1 / 32.

[0022] In combination with the first aspect, optionally, the first parameter enhancement module, the second parameter enhancement module, the third parameter enhancement module or the third parameter enhancement module respectively include: a 1×1 convolution layer, a 3×3 convolution layer, and a reparameterized block structure. The reparameterized block structure includes: a 3×3 convolution layer, a 1×1 convolution layer, a feature merging layer, a batch normalization layer and an activation function layer, wherein the 3×3 convolution layer and the 1×1 convolution layer are parallel, and the output features of the two convolution layers are used as inputs of the feature merging layer.

[0023] In combination with the first aspect, optionally, the multi-scale feature fusion component includes: a channel attention module, a spatial pixel attention module and a feature accumulation layer, and the output of the channel attention module serves as the input of the spatial pixel attention module.

[0024] In combination with the first aspect, optionally, the channel attention module is used to extract a channel feature map, and the steps include:

[0025] Calculate the attention weight matrix for each channel;

[0026] The Softmax function is used to normalize and calibrate the attention weight matrix of each channel to obtain a global dependency matrix of each channel;

[0027] Multiplying the global dependency matrix of each channel by the image information of the corresponding channel to obtain a feature map of each channel;

[0028] The feature maps of each channel are concatenated in parallel to obtain the channel feature map.

[0029] In combination with the first aspect, optionally, the spatial pixel attention module extracts a spatial pixel feature map based on the channel feature map, and the steps include:

[0030] Performing global average pooling on the channel feature map to obtain an average pooling feature map;

[0031] Performing global maximum pooling on the channel feature map to obtain a maximum pooling feature map;

[0032] Splicing the average pooling feature map and the maximum pooling feature map to obtain a spliced ​​feature map;

[0033] Convolving the concatenated feature map with a 3×3 convolution kernel and activating it with a Sigmoid function to obtain a spatial pixel attention weight matrix;

[0034] The spatial pixel attention weight matrix is ​​multiplied pixel by pixel with each channel feature matrix in the channel feature map to obtain the spatial pixel feature map.

[0035] In a second aspect, the present invention provides a dual-mode real-time small target detection device, comprising:

[0036] Image acquisition module: acquires the color image to be detected;

[0037] Segmentation and marking module: inputting the color image to be detected into an image segmentation network model to obtain a segmentation and marking image;

[0038] Target detection module: input the color image and the segmentation mark image into a pre-trained target detection and recognition network to obtain the category information and anchor frame information of the target to be detected;

[0039] The pre-trained target detection and recognition network includes: a dual-channel image fusion component, a re-parameterized enhanced feature extraction component, a multi-scale feature fusion component and a target prediction component.

[0040] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, it implements the dual-modal real-time small target detection method as described in any one of the first aspects.

[0041] In a fourth aspect, the present invention provides a device, characterized in that it includes:

[0042] A memory for storing instructions;

[0043] A processor is used to execute the instructions so that the device performs operations to implement the dual-modal real-time small target detection method as described in any one of the first aspects.

[0044] Compared with the prior art, the beneficial effect of a dual-modal real-time small target detection method of the present invention is that: the present invention obtains a color image of a target to be detected; inputs the color image of the target to be detected into an image segmentation network model to obtain a segmented marked image; inputs the color image and the segmented marked image into a pre-trained target detection and recognition network to obtain category information and anchor frame information of the target to be detected, thereby improving the detection accuracy. The fusion of dual-modal information can capture more features of small targets and reduce missed detections. At the same time, it can meet real-time requirements and can work stably in complex environments.

[0045] The small target detection device of the embodiment of the present application includes the small target detection method described above, and therefore can have all the technical features and technical effects of the small target detection method described above, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a flow chart of a dual-modal real-time small target detection method of the present invention;

[0048] Figure 2 The structure block diagram of a dual-mode real-time small target detection device of the present invention. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] In the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "top", "bottom", "one side", "the other side", "front", "back", "middle part", "inside", "top end", "bottom end", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention; the terms "first", "second", and "third" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance; in addition, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0051] Embodiment 1

[0052] like Figure 1 As shown, an embodiment of the present invention provides a dual-modal real-time small target detection method, including:

[0053] S1: Acquire a color image to be detected;

[0054] In a specific embodiment, an image acquisition device is started, which may be a high-resolution color camera. The camera is placed at a suitable position and angle to ensure that its field of view covers the area where the target may appear. This image contains rich color information, texture details, and brightness features, etc., which provides an important data basis for subsequent small target detection.

[0055] S2: inputting the color image into an image segmentation network model to obtain a segmentation mark image;

[0056] In a specific embodiment, the image segmentation network model includes: an encoder component, a self-attention and multi-layer perception component, a long-distance jump link component and a decoder component. Specifically, the encoder and the decoder are CNN structures for extracting local features to obtain better image representation. Transformer reflects complex spatial transformations and long-distance feature dependencies through self-attention and multi-layer perceptron (MLP) structures to obtain global feature representation. The three long-distance connections are inspired by UNet, combining low-level spatial information with high-level semantic information to achieve high-quality segmentation. A lightweight hole bottleneck is used in the encoder, and at the bottleneck, the number of channels of the input feature is reduced by half through 1×1 convolution. After reducing the number of channels, the number of parameters and the amount of calculation are greatly reduced. Although this will lose some accuracy, stacking two such modules will be more beneficial to make up for this loss. At the same time, due to the use of 1×1 convolution, the network depth must be deepened to obtain a larger receptive field. Therefore, after the 1×1 convolution, 3×1 and 1×3 decomposition convolutions are added to expand the receptive field to capture a wider range of contextual information. In addition, decomposition convolution is also based on considering the number of parameters and the amount of calculation. Similarly, in the following two-branch structure, both branches also use decomposed convolution, one of which is responsible for local and short-range feature information, and the other uses dilated convolution, which is responsible for extracting feature information from a larger receptive field at different dilation rates.

[0057] S3: Input the color image and the segmented marked image into a pre-trained target detection and recognition network to obtain category information and anchor frame information of the target to be detected; wherein the pre-trained target detection and recognition network includes: a dual-channel image fusion component, a re-parameterized enhanced feature extraction component, a multi-scale feature fusion component and a target prediction component.

[0058] In a specific embodiment, the dual-channel image fusion component is used for image fusion, and the steps include:

[0059] S3-1-1: normalizing the color image to obtain three-channel information of the normalized color image;

[0060] S3-1-2: Pixel-level splicing of the normalized three-channel information of the color image and the segmentation mark image information to obtain fused image information. The fused image information includes four channels, namely, the R channel, the G channel, the B channel of the color image and the mark value channel of the segmentation mark image.

[0061] In a specific embodiment, the re-parameterized enhanced feature extraction component includes: a first re-parameterized enhanced module, a second re-parameterized enhanced module, a third re-parameterized enhanced module, and a fourth re-parameterized enhanced module.

[0062] Among them, the output of the first parameterized enhancement module is used as the input of the second parameterized enhancement module, the output of the second parameterized enhancement module is used as the input of the third parameterized enhancement module, and the output of the third parameterized enhancement module is used as the input of the fourth parameterized enhancement module.

[0063] The image downsampling rate of the first parameterized enhancement module is 1, the image downsampling rate of the second parameterized enhancement module is 1 / 8, the image downsampling rate of the third parameterized enhancement module is 1 / 16, and the image downsampling rate of the fourth parameterized enhancement module is 1 / 32.

[0064] The first parameter enhancement module, the second parameter enhancement module, the third parameter enhancement module or the fourth parameter enhancement module respectively include: a 1×1 convolution layer, a 3×3 convolution layer, and a re-parameterized block structure, wherein the re-parameterized block structure includes: a 3×3 convolution layer, a 1×1 convolution layer, a feature merging layer, a batch normalization layer and an activation function layer, wherein the 3×3 convolution layer and the 1×1 convolution layer are parallel, and the output features of the two convolution layers are used as inputs of the feature merging layer.

[0065] The heavy parameter enhancement module in the present invention has more gradient propagation paths during training, can extract richer target features, and has better performance; during reasoning, its multi-branch paths can be equivalently merged into a single-branch path, thereby speeding up the reasoning speed without changing the model performance.

[0066] On the other hand, 3×3 convolution has stronger feature extraction capabilities than 1×1 convolution. Although replacing 1×1 convolution with 3×3 convolution will increase the number of model parameters and computation, computing libraries on modern GPUs and CPUs, such as NVIDIA's cuDNN and Intel's MKL, have been specially optimized for 3×3 convolution operations, and 3×3 convolution has a faster processing speed. Therefore, replacing 1×1 convolution with 3×3 convolution will not significantly increase the model inference time.

[0067] In a specific embodiment, the multi-scale feature fusion component includes: a channel attention module, a spatial pixel attention module and a feature accumulation layer, and the output of the channel attention module serves as the input of the spatial pixel attention module.

[0068] The channel attention module is used to extract the channel feature map, and the steps include:

[0069] S3-2-1: Calculate the attention weight matrix for each channel;

[0070] S3-2-2: Use the Softmax function to normalize and calibrate the attention weight matrix of each channel to obtain the global dependency matrix of each channel;

[0071] S3-2-3: multiplying the global dependency matrix of each channel by the image information of the corresponding channel to obtain a feature map of each channel;

[0072] S3-2-4: parallelly concatenate the feature maps of each channel to obtain the channel feature map.

[0073] In a specific embodiment, the spatial pixel attention module extracts a spatial pixel feature map based on the channel feature map, and the steps include:

[0074] S3-3-1: Perform global average pooling on the channel feature map to obtain an average pooling feature map;

[0075] S3-3-2: Perform global maximum pooling on the channel feature map to obtain a maximum pooling feature map;

[0076] S3-3-3: splicing the average pooling feature map and the maximum pooling feature map to obtain a spliced ​​feature map;

[0077] S3-3-4: convolve the concatenated feature map using a 3×3 convolution kernel and activate it through a Sigmoid function to obtain a spatial pixel attention weight matrix;

[0078] S3-3-5: Multiply the spatial pixel attention weight matrix by each channel feature matrix in the channel feature map pixel by pixel to obtain the spatial pixel feature map.

[0079] The present invention provides a dual-modal real-time small target detection method, which improves detection accuracy, and can capture more features of small targets by fusing dual-modal information, reduce missed detections, meet real-time requirements, and work stably in complex environments.

[0080] Embodiment 2

[0081] like Figure 2As shown, an embodiment of the present invention provides a dual-mode real-time small target detection device, comprising:

[0082] Image acquisition module: acquires the color image to be detected;

[0083] Segmentation and marking module: inputting the color image to be detected into an image segmentation network model to obtain a segmentation and marking image;

[0084] Target detection module: input the color image and the segmentation mark image into a pre-trained target detection and recognition network to obtain the category information and anchor frame information of the target to be detected;

[0085] The pre-trained target detection and recognition network includes: a dual-channel image fusion component, a re-parameterized enhanced feature extraction component, a multi-scale feature fusion component and a target prediction component.

[0086] The present invention provides a dual-modal real-time small target detection device, which improves detection accuracy, can capture more features of small targets by fusing dual-modal information, and reduce missed detections. At the same time, it can meet real-time requirements and can work stably in complex environments.

[0087] Embodiment 3

[0088] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a dual-modal real-time small target detection method as described in Embodiment 1 is implemented.

[0089] Embodiment 4:

[0090] An embodiment of the present invention provides a device, including:

[0091] A memory for storing instructions;

[0092] The processor is used to execute the instruction so that the device performs the operation of implementing the dual-modal real-time small target detection method as described in any one of the embodiments.

[0093] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0094] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0097] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A dual-modal real-time small target detection method, characterized in that: include: Acquire a color image to be detected; Inputting the color image into an image segmentation network model to obtain a segmentation marked image; Inputting the color image to be detected and the segmented marked image into a pre-trained target detection and recognition network to obtain category information and anchor frame information of the target to be detected; The pre-trained target detection and recognition network is composed of a dual-channel image fusion component, a re-parameterized enhanced feature extraction component, a multi-scale feature fusion component and a target prediction component.

2. A dual-modal real-time small target detection method according to claim 1, characterized in that: The image segmentation network model includes an encoder component, a self-attention and multi-layer perception component, a long-distance skip link component and a decoder component.

3. The dual-modal real-time small target detection method according to claim 1, characterized in that: The dual-channel image fusion component is used for image fusion, and the steps include: Normalizing the color image to obtain normalized three-channel information of the color image; Performing pixel-level splicing on the normalized three-channel information of the color image and the segmentation mark image information to obtain fused image information; The fused image information includes four channels, namely, an R channel, a G channel, a B channel of the color image and a label value channel of the segmented label image.

4. The dual-modal real-time small target detection method according to claim 1, characterized in that: The re-parameterized enhanced feature extraction component comprises a first re-parameterized enhanced module, a second re-parameterized enhanced module, a third re-parameterized enhanced module and a fourth re-parameterized enhanced module; Among them, the output of the first parameterized enhancement module is used as the input of the second parameterized enhancement module, the output of the second parameterized enhancement module is used as the input of the third parameterized enhancement module, and the output of the third parameterized enhancement module is used as the input of the fourth parameterized enhancement module.

5. A dual-modal real-time small target detection method according to claim 4, characterized in that: The image downsampling rate of the first parameterized enhancement module is 1; the image downsampling rate of the second parameterized enhancement module is 1 / 8; the image downsampling rate of the third parameterized enhancement module is 1 / 16; and the image downsampling rate of the fourth parameterized enhancement module is 1 / 32.

6. A dual-modal real-time small target detection method according to claim 4 or 5, characterized in that: The first parameter enhancement module, the second parameter enhancement module, the third parameter enhancement module or the fourth parameter enhancement module respectively comprises: a 1×1 convolution layer, a 3×3 convolution layer, and a re-parameterized block structure; The re-parameterized block structure includes: a 3×3 convolution layer, a 1×1 convolution layer, a feature merging layer, a batch normalization layer and an activation function layer; The 3×3 convolution layer and the 1×1 convolution layer of the re-parameterized block structure are in parallel, and the output features of the two convolution layers are used as inputs of the feature merging layer.

7. A dual-modal real-time small target detection method according to claim 1, characterized in that: The multi-scale feature fusion component includes a channel attention module, a spatial pixel attention module and a feature accumulation layer; The output of the channel attention module serves as the input of the spatial pixel attention module.

8. A dual-modal real-time small target detection method according to claim 7, characterized in that: The channel attention module is used to extract the channel feature map, and the steps include: Calculate the attention weight matrix for each channel; The Softmax function is used to normalize and calibrate the attention weight matrix of each channel to obtain a global dependency matrix of each channel; Multiplying the global dependency matrix of each channel by the image information of the corresponding channel to obtain a feature map of each channel; The feature maps of each channel are spliced ​​in parallel to obtain the channel feature map.

9. A dual-modal real-time small target detection method according to claim 7, characterized in that: The spatial pixel attention module extracts a spatial pixel feature map based on the channel feature map, and the steps include: Performing global average pooling on the channel feature map to obtain an average pooling feature map; Performing global maximum pooling on the channel feature map to obtain a maximum pooling feature map; Concatenating the average pooling feature map and the maximum pooling feature map to obtain a concatenated feature map; Convolving the concatenated feature map with a 3×3 convolution kernel and activating it with a Sigmoid function to obtain a spatial pixel attention weight matrix; The spatial pixel attention weight matrix is ​​multiplied pixel by pixel with each channel feature matrix in the channel feature map to obtain the spatial pixel feature map.

10. A dual-mode real-time small target detection device, characterized in that: include: An image acquisition module, which acquires a color image to be detected; A segmentation and marking module, inputting the color image to be detected into an image segmentation network model to obtain a segmentation and marking image; The target detection module inputs the color image and the segmentation mark image into a pre-trained target detection and recognition network to obtain the category information and anchor frame information of the target to be detected; The pre-trained target detection and recognition network includes: a dual-channel image fusion component, a re-parameterized enhanced feature extraction component, a multi-scale feature fusion component and a target prediction component.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the dual-modal real-time small target detection method as described in any one of claims 1 to 9 is implemented.

12. A device, characterized in that: include: A memory for storing instructions; A processor is used to execute the instruction so that the device performs the operation of implementing the dual-modal real-time small target detection method as described in any one of claims 1-9.

Images