Method and device for detecting small target in dark environment
By building a brightness enhancement network and a multi-scale detection network, using specific network structures and modules, the problem of low detection accuracy of small objects in dim environments is solved, and higher detection accuracy and lower missed detection rate are achieved.
Patent Information
- Application Number
- CN202411433840.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2025-03-11
AI Technical Summary
The prior art is difficult to extract rich feature information in dim environments, resulting in high detection rate of small targets and low detection accuracy.
By building a brightness enhancement network, backbone network and multi-scale detection network, using downsampling layer, upsampling layer and jump connection structure, combining inverse residual blocks and multi-layer residual units, a small object detection model is built, brightness enhancement and feature extraction is performed, and multi-scale features are fused to improve detection accuracy.
It effectively reduces the missed detection rate of small targets in dim environments, improves detection accuracy, and solves the problem of low detection accuracy in the prior art.
Smart Images
Figure CN119600325A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of target detection technology, and in particular to a method and device for detecting small targets in a dim environment. Background Art
[0002] With the rapid development of Convolution Neural Network (CNN), deep learning has been widely used in the field of image recognition. The current mainstream target detection algorithm is the YOLO series algorithm, which can simultaneously predict the bounding box coordinates and the probability of the category to which it belongs. Since the algorithm has strong generalization ability, many researchers have improved it and it has been widely used in the field of target detection. In recent years, the YOLO algorithm has been continuously improved, and the network has been continuously lightweight, which has continuously improved the accuracy of the target. However, under dim conditions, it is difficult for the above method to extract rich feature information, and misidentification will occur between targets with great similarity. In addition, since the resolution of small targets is low and the detail information is not prominent, the above method may miss detection when detecting small targets, resulting in low detection accuracy. Summary of the invention
[0003] In view of this, the embodiments of the present application provide a method, device, electronic device and computer-readable storage medium for detecting small targets in a dim environment to solve the problems of high missed detection rate and low detection accuracy of small targets in a dim environment in the prior art.
[0004] According to a first aspect of an embodiment of the present application, a method for detecting small targets in a dim environment is provided, comprising: constructing a brightness enhancement network using downsampling layers, upsampling layers and a jump connection structure, constructing a multi-layer residual unit using different numbers of inverse residual blocks, constructing a backbone network using multi-layer residual units, and constructing a multi-scale detection network using upsampling layers, downsampling layers, addition layers, decoupling heads and non-maximum suppression layers; constructing a small target detection model using the brightness enhancement network, the backbone network and the multi-scale detection network; acquiring a training image, and inputting the training image into the small target detection model: performing brightness enhancement on the training image through the brightness enhancement network to obtain an enhanced image; extracting multiple scale features of the enhanced image through residual units of different layers in the backbone network; fusing multiple scale features through the multi-scale detection network, and determining a training detection result based on the fusion result; calculating a loss based on the training image, the enhanced image, the training detection result and the label of the training image, and optimizing the model parameters of the small target detection model based on the loss to complete the training of the small target detection model.
[0005] According to a second aspect of an embodiment of the present application, a small target detection device in a dim environment is provided, comprising: a first construction module, configured to construct a brightness enhancement network using a downsampling layer, an upsampling layer and a jump connection structure, construct a multi-layer residual unit using different numbers of inverse residual blocks, construct a backbone network using the multi-layer residual unit, and construct a multi-scale detection network using the upsampling layer, the downsampling layer, the addition layer, the decoupling head and the non-maximum suppression layer; a second construction module, configured to construct a small target detection model using the brightness enhancement network, the backbone network and the multi-scale detection network; an enhancement module, configured to obtain a training image, input the training image into the small target detection model: perform brightness enhancement on the training image through the brightness enhancement network to obtain an enhanced image; an extraction module, configured to extract multiple scale features of the enhanced image through the residual units of different layers in the backbone network; a fusion module, configured to fuse multiple scale features through the multi-scale detection network, and determine the training detection result based on the fusion result; an optimization module, configured to calculate the loss based on the training image, the enhanced image, the training detection result and the label of the training image, and optimize the model parameters of the small target detection model based on the loss to complete the training of the small target detection model.
[0006] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0007] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0008] Compared with the prior art, the embodiments of the present application have the following beneficial effects: constructing a brightness enhancement network using downsampling layers, upsampling layers and a jump connection structure, constructing a multi-layer residual unit using different numbers of inverse residual blocks, constructing a backbone network using multi-layer residual units, and constructing a multi-scale detection network using upsampling layers, downsampling layers, addition layers, decoupling heads and non-maximum suppression layers; constructing a small target detection model using the brightness enhancement network, the backbone network and the multi-scale detection network; obtaining a training image, and inputting the training image into the small target detection model: performing brightness enhancement on the training image through the brightness enhancement network to obtain an enhanced image; extracting multiple scale features of the enhanced image through the residual units of different layers in the backbone network; fusing multiple scale features through the multi-scale detection network, and determining the training detection result based on the fusion result; calculating the loss based on the training image, the enhanced image, the training detection result and the label of the training image, and optimizing the model parameters of the small target detection model based on the loss to complete the training of the small target detection model. The above technical means can solve the problems of high missed detection rate of small targets in dim environments and low detection accuracy in the prior art, thereby reducing the missed detection rate of small targets in dim environments and improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0010] Figure 1 It is a flowchart of a small target detection method in a dim environment provided by an embodiment of the present application;
[0011] Figure 2 is a flow chart of another small target detection method in a dim environment provided by an embodiment of the present application;
[0012] Figure 3 This is a schematic diagram of the structure of a small target detection device in a dim environment provided by an embodiment of the present application;
[0013] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0014] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0015] A method and device for detecting small targets in a dim environment according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0016] Figure 1 It is a flowchart of a small target detection method in a dim environment provided in an embodiment of the present application. Figure 1 The small target detection method in a dim environment can be executed by a computer or a server, or by software on the computer or the server. Figure 1 As shown, the small target detection method in a dim environment includes:
[0017] S101, construct a brightness enhancement network using downsampling layers, upsampling layers and skip connection structures, construct multi-layer residual units using different numbers of inverse residual blocks, construct a backbone network using multi-layer residual units, and construct a multi-scale detection network using upsampling layers, downsampling layers, addition layers, decoupling heads and non-maximum suppression layers;
[0018] S102, construct a small target detection model using the brightness enhancement network, backbone network and multi-scale detection network;
[0019] S103, obtaining a training image, and inputting the training image into a small target detection model: performing brightness enhancement on the training image through a brightness enhancement network to obtain an enhanced image;
[0020] S104, extracting multiple scale features of the enhanced image through residual units of different layers in the backbone network;
[0021] S105, fusing multiple scale features through a multi-scale detection network, and determining a training detection result based on the fusion result;
[0022] S106, calculating the loss based on the training image, the enhanced image, the training detection result and the label of the training image, and optimizing the model parameters of the small target detection model based on the loss to complete the training of the small target detection model.
[0023] The downsampling layer is a network layer that performs downsampling processing, the upsampling layer is a network layer that performs upsampling processing, and the skip connection structure connects non-adjacent network layers together and performs corresponding processing (such as addition). The inverse residual block adds an expansion layer and a depth-separable convolution to the residual block to reduce the number of parameters and computational cost. The addition layer adds the inputs of multiple addition layers and outputs them. The decoupling head is a classification network that generally includes a fully connected layer and an activation function layer. The non-maximum suppression layer is a Non-Maximum Suppression layer that is used to select the bounding box that is most likely to contain the target object from a series of candidate areas and suppress overlapping candidate boxes with low confidence.
[0024] The training images are images of small-sized targets acquired in dim environments. The brightness enhancement network enhances the brightness of the training images (to cope with dim environments), the residual units at different layers in the backbone network extract multiple scale features of the enhanced images (to cope with small-sized targets), and the multi-scale detection network fuses multiple scale features and determines the training detection results.
[0025] According to the technical solution provided by the embodiment of the present application, a brightness enhancement network is constructed using a downsampling layer, an upsampling layer and a jump connection structure, a multi-layer residual unit is constructed using different numbers of inverse residual blocks, a backbone network is constructed using multi-layer residual units, and a multi-scale detection network is constructed using upsampling layers, downsampling layers, addition layers, decoupling heads and non-maximum suppression layers; a small target detection model is constructed using the brightness enhancement network, the backbone network and the multi-scale detection network; a training image is obtained, and the training image is input into the small target detection model: the training image is brightness enhanced by the brightness enhancement network to obtain an enhanced image; multiple scale features of the enhanced image are extracted by residual units of different layers in the backbone network; multiple scale features are fused by the multi-scale detection network, and the training detection result is determined based on the fusion result; the loss is calculated based on the training image, the enhanced image, the training detection result and the label of the training image, and the model parameters of the small target detection model are optimized based on the loss to complete the training of the small target detection model. The above technical means can solve the problems of high small target missed detection rate and low detection accuracy in dim environments in the prior art, thereby reducing the small target missed detection rate in dim environments and improving the detection accuracy.
[0026] Furthermore, a brightness enhancement network is constructed using downsampling layers, upsampling layers and skip connection structures, including: sequentially connecting convolutional layers, batch layers and activation layers to obtain a nonlinear mapping layer; sequentially connecting N downsampling layers and N upsampling layers after the nonlinear mapping layer, and setting a skip connection structure between the i-th downsampling layer and the N-i+1 downsampling layers to obtain a brightness enhancement network, wherein the skip connection structure is used to add the output of the downsampling layer to the input of the upsampling layer.
[0027] A convolutional layer, a batch layer, and an activation layer are connected in sequence to obtain a nonlinear mapping layer. N is a positive integer, the number of downsampling layers and upsampling layers is N, and i is the serial number of the downsampling layer.
[0028] Furthermore, the brightness of the training image is enhanced by a brightness enhancement network to obtain an enhanced image, including: N is 3, and inside the brightness enhancement network: the training image is processed by a nonlinear mapping layer to obtain image features; the image features are sequentially passed through three downsampling layers to obtain a first downsampling feature, a second downsampling feature, and a third downsampling feature; the third downsampling feature is processed by the first upsampling layer to obtain a first upsampling feature, and the first upsampling feature and the third downsampling feature are added through a jump connection structure between the third downsampling layer and the first upsampling layer to obtain a third upsampling feature. a sampling feature; processing the first sampling feature through a second upsampling layer to obtain a second upsampling feature, adding the second upsampling feature and the second downsampling feature through a skip connection structure between the second downsampling layer and the second upsampling layer to obtain a second sampling feature; processing the second sampling feature through a third upsampling layer to obtain a third upsampling feature, adding the third upsampling feature and the first downsampling feature through a skip connection structure between the third downsampling layer and the first upsampling layer to obtain a third sampling feature; adding the third sampling feature and the training image to obtain an enhanced image.
[0029] Furthermore, the brightness of the training image is enhanced by a brightness enhancement network to obtain an enhanced image, including: N is 2, and inside the brightness enhancement network: the training image is processed by a nonlinear mapping layer to obtain image features; the image features are passed through two downsampling layers in sequence to obtain a first downsampling feature and a second downsampling feature; the second downsampling feature is processed by the first upsampling layer to obtain a first upsampling feature, and the first upsampling feature and the second downsampling feature are added through a jump connection structure between the second downsampling layer and the first upsampling layer to obtain a first sampling feature; the first sampling feature is processed by the second upsampling layer to obtain a second upsampling feature, and the second upsampling feature and the first downsampling feature are added through a jump connection structure between the first downsampling layer and the second upsampling layer to obtain a second sampling feature; the second sampling feature and the training image are added to obtain an enhanced image.
[0030] The image is composed of pixels, so the image itself is a feature, so the third sampling feature can be directly added to the training image, and the second sampling feature can be directly added to the training image.
[0031] Furthermore, multiple scale features of the enhanced image are extracted through residual units of different layers in the backbone network, including: the backbone network includes 5 layers of residual units, the numbers of inverse residual blocks in the 5 layers of residual units are 1, 2, 6, 6 and 4 respectively, and inside the backbone network: the enhanced image is processed by the first layer of residual units and the second layer of residual units in turn to obtain a first scale feature; the first scale feature is processed by the third layer of residual units to obtain a second scale feature; the second scale feature is processed by the fourth layer of residual units to obtain a third scale feature; the third scale feature is processed by the fifth layer of residual units to obtain a fourth scale feature.
[0032] The enhanced image is processed by the first-layer residual unit and the second-layer residual unit in sequence, which is the result of the second-layer residual unit processing the enhanced image processed by the first-layer residual unit. The result of the second-layer residual unit processing is the first scale feature.
[0033] The multi-scale detection network can be seen as consisting of 6 feature fusion networks, 1 decoupling head and 1 non-maximum suppression layer. Each feature fusion network is composed of an upsampling layer or a downsampling layer and an addition layer. The 6 feature fusion networks are densely connected.
[0034] Furthermore, multiple scale features are fused through a multi-scale detection network, and a training detection result is determined based on the fusion result, including: in the first feature fusion network: the fourth scale feature is upsampled through the upsampling layer, and the fourth scale feature after the upsampling process is added to the third scale feature through the addition layer to obtain a first added feature; in the second feature fusion network: the first added feature and the fourth scale feature are upsampled through the upsampling layer, and the first added feature and the fourth scale feature after the upsampling process are added to the second scale feature through the addition layer to obtain a second added feature; in the third feature fusion network: the first added feature, the second added feature and the fourth scale feature are upsampled through the upsampling layer, and the first added feature, the second added feature and the fourth scale feature after the upsampling process are added to the first scale feature through the addition layer to obtain a third added feature; In the fourth feature fusion network: the third added feature is downsampled through the downsampling layer, and the third added feature after the downsampling process is added to the second scale feature through the addition layer to obtain the fourth added feature; in the fifth feature fusion network: the fourth added feature is downsampled through the downsampling layer, and the fourth added feature after the downsampling process is added to the third scale feature through the addition layer to obtain the fifth added feature; in the sixth feature fusion network: the fifth added feature is downsampled through the downsampling layer, and the fifth added feature after the downsampling process is added to the fourth scale feature through the addition layer to obtain the sixth added feature; the third added feature, the fifth added feature and the sixth added feature are processed respectively by the decoupling head to obtain the first detection result, the second detection result and the third detection result; the first detection result, the second detection result and the third detection result are processed by the non-maximum suppression layer to obtain the training detection result.
[0035] The non-maximum suppression layer selects the bounding box that is most likely to contain the target object from the first detection result, the second detection result, and the third detection result, and suppresses the overlapping candidate boxes with low confidence to obtain the training detection result.
[0036] Furthermore, the loss is calculated based on the training image, the enhanced image, the training detection result and the label of the training image, including: using the minimum absolute deviation loss function to calculate the enhancement loss between the training image and the enhanced image; using the cross entropy loss function to calculate the detection loss between the training detection result and the label of the training image.
[0037] In an optional embodiment, the pixels in the enhanced image are arranged from large to small according to the brightness value, the last part is selected as the dim area in a certain proportion, and the rest is selected as the dark area, and finally the two parts are weighted according to the required brightness enhancement strength (the weight assigned to the large brightness is small, and the weight assigned to the small brightness is large), and the enhanced image is adjusted according to the weight assigned. The above technical means can ensure that the brightness of the enhanced image is uniform, rather than some parts being particularly bright and some parts being particularly dark.
[0038] The enhancement loss between the training image and the adjusted augmented image is calculated using the minimum absolute deviation loss function.
[0039] Furthermore, the model parameters of the small target detection model are optimized according to the loss to complete the training of the small target detection model, including: optimizing the model parameters of the brightness enhancement network according to the enhancement loss; optimizing the model parameters of the backbone network and the multi-scale detection network according to the detection loss.
[0040] In an optional embodiment, the model parameters of the backbone network and the multi-scale detection network are frozen, and the model parameters of the brightness enhancement network are optimized based on the enhancement loss; the model parameters of the brightness enhancement network are frozen, and the model parameters of the backbone network and the multi-scale detection network are optimized based on the detection loss.
[0041] Figure 2 FIG. 1 is a flow chart of another small target detection method in a dim environment provided by an embodiment of the present application. Figure 2 As shown, the method includes:
[0042] S201, obtain the target image to be detected, and input the target image into the small target detection model:
[0043] S202, performing brightness enhancement on the target image through a brightness enhancement network to obtain a target enhanced image;
[0044] S203, extracting multiple scale features of the target enhanced image through residual units of different layers in the backbone network;
[0045] S204, fusing multiple scale features of the target enhanced image through a multi-scale detection network, and determining a target detection result based on the fusion result.
[0046] The embodiment of the present application is the reasoning stage of the small target detection model, and the embodiments before the embodiment of the present application are the training stage of the small target detection model. Because the training stage and the reasoning stage correspond to each other, the reasoning stage will not be described in detail here.
[0047] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0048] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0049] Figure 3 Schematic diagram of a small target detection device in a dim environment provided by an embodiment of the present application. Figure 3 As shown, the small target detection device in a dim environment includes:
[0050] The first construction module 301 is configured to construct a brightness enhancement network using a downsampling layer, an upsampling layer and a skip connection structure, construct a multi-layer residual unit using different numbers of inverse residual blocks, construct a backbone network using multi-layer residual units, and construct a multi-scale detection network using an upsampling layer, a downsampling layer, an addition layer, a decoupling head and a non-maximum suppression layer;
[0051] The second construction module 302 is configured to construct a small target detection model using the brightness enhancement network, the backbone network and the multi-scale detection network;
[0052] The enhancement module 303 is configured to obtain a training image, input the training image into the small target detection model, and perform brightness enhancement on the training image through a brightness enhancement network to obtain an enhanced image;
[0053] An extraction module 304 is configured to extract multiple scale features of the enhanced image through residual units of different layers in the backbone network;
[0054] A fusion module 305 is configured to fuse multiple scale features through a multi-scale detection network and determine a training detection result based on the fusion result;
[0055] The optimization module 306 is configured to calculate the loss based on the training image, the enhanced image, the training detection result and the label of the training image, and optimize the model parameters of the small target detection model based on the loss to complete the training of the small target detection model.
[0056] According to the technical solution provided by the embodiment of the present application, a brightness enhancement network is constructed using a downsampling layer, an upsampling layer and a jump connection structure, a multi-layer residual unit is constructed using different numbers of inverse residual blocks, a backbone network is constructed using multi-layer residual units, and a multi-scale detection network is constructed using upsampling layers, downsampling layers, addition layers, decoupling heads and non-maximum suppression layers; a small target detection model is constructed using the brightness enhancement network, the backbone network and the multi-scale detection network; a training image is obtained, and the training image is input into the small target detection model: the training image is brightness enhanced by the brightness enhancement network to obtain an enhanced image; multiple scale features of the enhanced image are extracted by residual units of different layers in the backbone network; multiple scale features are fused by the multi-scale detection network, and the training detection result is determined based on the fusion result; the loss is calculated based on the training image, the enhanced image, the training detection result and the label of the training image, and the model parameters of the small target detection model are optimized based on the loss to complete the training of the small target detection model. The above technical means can solve the problems of high small target missed detection rate and low detection accuracy in dim environments in the prior art, thereby reducing the small target missed detection rate in dim environments and improving the detection accuracy.
[0057] In some embodiments, the first building module 301 is further configured to sequentially connect a convolutional layer, a batch layer, and an activation layer to obtain a nonlinear mapping layer; sequentially connect N downsampling layers and N upsampling layers after the nonlinear mapping layer, and set a jump connection structure between the i-th downsampling layer and the N-i+1 downsampling layers to obtain a brightness enhancement network, wherein the jump connection structure is used to add the output of the downsampling layer to the input of the upsampling layer.
[0058] In some embodiments, the enhancement module 303 is further configured such that N is 3, and inside the brightness enhancement network: the training image is processed through a nonlinear mapping layer to obtain image features; the image features are sequentially passed through three down-sampling layers to obtain a first down-sampling feature, a second down-sampling feature, and a third down-sampling feature; the third down-sampling feature is processed through the first up-sampling layer to obtain a first up-sampling feature, and the first up-sampling feature and the third down-sampling feature are added through a skip connection structure between the third down-sampling layer and the first up-sampling layer to obtain a first sampling feature; the first sampling feature is processed through the second up-sampling layer to obtain a second up-sampling feature, and the second up-sampling feature and the second down-sampling feature are added through a skip connection structure between the second down-sampling layer and the second up-sampling layer to obtain a second sampling feature; the second sampling feature is processed through the third up-sampling layer to obtain a third up-sampling feature, and the third up-sampling feature and the first down-sampling feature are added through a skip connection structure between the third down-sampling layer and the first up-sampling layer to obtain a third sampling feature; the third sampling feature and the training image are added to obtain an enhanced image.
[0059] In some embodiments, the enhancement module 303 is also configured as N is 2, and inside the brightness enhancement network: the training image is processed through a nonlinear mapping layer to obtain image features; the image features are passed through two downsampling layers in sequence to obtain a first downsampling feature and a second downsampling feature; the second downsampling feature is processed through the first upsampling layer to obtain a first upsampling feature, and the first upsampling feature and the second downsampling feature are added through a jump connection structure between the second downsampling layer and the first upsampling layer to obtain a first sampling feature; the first sampling feature is processed through the second upsampling layer to obtain a second upsampling feature, and the second upsampling feature and the first downsampling feature are added through a jump connection structure between the first downsampling layer and the second upsampling layer to obtain a second sampling feature; the second sampling feature and the training image are added to obtain an enhanced image.
[0060] In some embodiments, the extraction module 304 is further configured to include 5 layers of residual units in the backbone network, and the numbers of inverse residual blocks in the 5 layers of residual units are 1, 2, 6, 6 and 4 respectively. Inside the backbone network: the enhanced image is processed by the 1st layer residual unit and the 2nd layer residual unit in turn to obtain the first scale feature; the first scale feature is processed by the 3rd layer residual unit to obtain the second scale feature; the second scale feature is processed by the 4th layer residual unit to obtain the third scale feature; the third scale feature is processed by the 5th layer residual unit to obtain the fourth scale feature.
[0061] In some embodiments, the fusion module 305 is further configured to: in the first feature fusion network: upsample the fourth scale feature through the upsampling layer, add the fourth scale feature after the upsampling process and the third scale feature through the addition layer to obtain a first added feature; in the second feature fusion network: upsample the first added feature and the fourth scale feature through the upsampling layer, add the first added feature and the fourth scale feature after the upsampling process to the second scale feature through the addition layer to obtain a second added feature; in the third feature fusion network: upsample the first added feature, the second added feature and the fourth scale feature through the upsampling layer, add the first added feature, the second added feature and the fourth scale feature after the upsampling process to the first scale feature through the addition layer to obtain a third added feature; in the fourth feature fusion network: The third added feature is downsampled through the downsampling layer, and the third added feature after the downsampling process is added to the second scale feature through the addition layer to obtain a fourth added feature; in the fifth feature fusion network: the fourth added feature is downsampled through the downsampling layer, and the fourth added feature after the downsampling process is added to the third scale feature through the addition layer to obtain a fifth added feature; in the sixth feature fusion network: the fifth added feature is downsampled through the downsampling layer, and the fifth added feature after the downsampling process is added to the fourth scale feature through the addition layer to obtain a sixth added feature; the third added feature, the fifth added feature and the sixth added feature are respectively processed through the decoupling head to obtain a first detection result, a second detection result and a third detection result; the first detection result, the second detection result and the third detection result are processed through the non-maximum suppression layer to obtain a training detection result.
[0062] In some embodiments, the optimization module 306 is further configured to calculate the enhancement loss between the training image and the enhanced image using a minimum absolute deviation loss function; and calculate the detection loss between the training detection result and the label of the training image using a cross entropy loss function.
[0063] In some embodiments, the enhancement module 303 is further configured to arrange the pixels in the enhanced image from large to small according to the brightness value, select the last part as the dim area and the rest as the dark area in a certain proportion, and finally assign weights to the two parts according to the required brightness enhancement strength (the weight assigned to the area with large brightness is small, and the weight assigned to the area with small brightness is large), and adjust the enhanced image according to the assigned weights. The above technical means can ensure that the brightness of the enhanced image is uniform, rather than some parts being particularly bright and some parts being particularly dark.
[0064] In some embodiments, the optimization module 306 is further configured to calculate the enhancement loss between the training image and the adjusted enhanced image using a minimum absolute deviation loss function.
[0065] In some embodiments, the optimization module 306 is further configured to optimize the model parameters of the brightness enhancement network according to the enhancement loss; and optimize the model parameters of the backbone network and the multi-scale detection network according to the detection loss.
[0066] In some embodiments, the optimization module 306 is further configured to freeze the model parameters of the backbone network and the multi-scale detection network, and optimize the model parameters of the brightness enhancement network based on the enhancement loss; freeze the model parameters of the brightness enhancement network, and optimize the model parameters of the backbone network and the multi-scale detection network based on the detection loss.
[0067] In some embodiments, the optimization module 306 is further configured to obtain a target image to be detected, input the target image into a small target detection model: perform brightness enhancement on the target image through a brightness enhancement network to obtain a target enhanced image; extract various scale features of the target enhanced image through residual units of different layers in the backbone network; fuse the various scale features of the target enhanced image through a multi-scale detection network, and determine the target detection result based on the fusion result.
[0068] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0069] Figure 4 Schematic diagram of an electronic device 4 provided in an embodiment of the present application. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0070] The electronic device 4 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 The electronic device 4 is merely an example and does not limit the electronic device 4 , and may include more or less components than those shown in the figure, or different components.
[0071] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0072] The memory 402 may be an internal storage unit of the electronic device 4, for example, a hard disk or memory of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. The memory 402 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.
[0073] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0074] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.
[0075] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A small target detection method in a dim environment, characterized in that: include: A brightness enhancement network is constructed using downsampling layers, upsampling layers and skip connection structures. Multi-layer residual units are constructed using different numbers of inverse residual blocks. A backbone network is constructed using multi-layer residual units. A multi-scale detection network is constructed using upsampling layers, downsampling layers, addition layers, decoupling heads and non-maximum suppression layers. Constructing a small target detection model using the brightness enhancement network, the backbone network and the multi-scale detection network; Obtain a training image and input the training image into the small target detection model: Performing brightness enhancement on the training image through the brightness enhancement network to obtain an enhanced image; Extracting multiple scale features of the enhanced image through residual units of different layers in the backbone network; fusing multiple scale features through the multi-scale detection network, and determining a training detection result based on the fusion result; The loss is calculated based on the training image, the enhanced image, the training detection result and the label of the training image, and the model parameters of the small target detection model are optimized based on the loss to complete the training of the small target detection model.
2. The method according to claim 1, characterized in that The brightness enhancement network is constructed using downsampling layers, upsampling layers and skip connection structures, including: Connect the convolutional layer, batch layer and activation layer in sequence to obtain a nonlinear mapping layer; N downsampling layers and N upsampling layers are connected in sequence after the nonlinear mapping layer, and the jump connection structure is set between the i-th downsampling layer and the N-i+1 downsampling layer to obtain the brightness enhancement network, wherein the jump connection structure is used to add the output of the downsampling layer to the input of the upsampling layer.
3. The method according to claim 2, characterized in that Performing brightness enhancement on the training image through the brightness enhancement network to obtain an enhanced image includes: N is 3, inside the brightness enhancement network: Processing the training image through the nonlinear mapping layer to obtain image features; Passing the image features through three downsampling layers in sequence to obtain a first downsampling feature, a second downsampling feature, and a third downsampling feature; Processing the third down-sampling feature through the first up-sampling layer to obtain a first up-sampling feature, and adding the first up-sampling feature and the third down-sampling feature through a skip connection structure between the third down-sampling layer and the first up-sampling layer to obtain a first sampling feature; Processing the first sampling feature through a second upsampling layer to obtain a second upsampling feature, and adding the second upsampling feature to the second downsampling feature through a skip connection structure between the second downsampling layer and the second upsampling layer to obtain a second sampling feature; Processing the second sampling feature through a third upsampling layer to obtain a third upsampling feature, and adding the third upsampling feature to the first downsampling feature through a skip connection structure between the third downsampling layer and the first upsampling layer to obtain a third sampling feature; The third sampling feature is added to the training image to obtain the enhanced image.
4. The method according to claim 1, characterized in that: Extracting multiple scale features of the enhanced image through residual units of different layers in the backbone network includes: The backbone network includes 5 layers of residual units, the numbers of inverse residual blocks in the 5 layers of residual units are 1, 2, 6, 6 and 4 respectively. Inside the backbone network: Processing the enhanced image through the first layer residual unit and the second layer residual unit in sequence to obtain a first scale feature; The first scale feature is processed by the third layer residual unit to obtain the second scale feature; The second scale feature is processed by the 4th layer residual unit to obtain the third scale feature; The third scale feature is processed by the fifth layer of residual unit to obtain the fourth scale feature.
5. The method according to claim 4, characterized in that Fusion of multiple scale features through the multi-scale detection network and determination of training detection results based on the fusion results include: The fourth scale feature is upsampled by the upsampling layer, and the fourth scale feature after the upsampling is added to the third scale feature by the addition layer to obtain a first added feature; The first added feature and the fourth scale feature are upsampled by the upsampling layer, and the first added feature and the fourth scale feature after the upsampling process are added to the second scale feature by the addition layer to obtain a second added feature; The first added feature, the second added feature and the fourth scale feature are upsampled by the upsampling layer, and the first added feature, the second added feature and the fourth scale feature after the upsampling are added to the first scale feature by the addition layer to obtain a third added feature; The third added feature is downsampled through the downsampling layer, and the third added feature after the downsampling process is added to the second scale feature through the addition layer to obtain a fourth added feature; The fourth added feature is downsampled through the downsampling layer, and the fourth added feature after the downsampling process and the third scale feature are added through the addition layer to obtain a fifth added feature; The fifth added feature is downsampled through the downsampling layer, and the fifth added feature after the downsampling process and the fourth scale feature are added through the addition layer to obtain a sixth added feature; The third added feature, the fifth added feature and the sixth added feature are processed respectively by the decoupling head to obtain a first detection result, a second detection result and a third detection result; The first detection result, the second detection result and the third detection result are processed by a non-maximum suppression layer to obtain the training detection result.
6. The method according to claim 1, characterized in that Calculating the loss according to the training image, the enhanced image, the training detection result, and the label of the training image, including: Calculating the enhancement loss between the training image and the enhanced image using a minimum absolute deviation loss function; The detection loss between the training detection result and the label of the training image is calculated using a cross entropy loss function.
7. The method according to claim 6, characterized in that Optimizing the model parameters of the small target detection model according to the loss to complete the training of the small target detection model, including: Optimizing the model parameters of the brightness enhancement network according to the enhancement loss; The detection loss optimizes model parameters of the backbone network and the multi-scale detection network.
8. A small target detection device in a dim environment, characterized in that: include: The first building module is configured to construct a brightness enhancement network using a downsampling layer, an upsampling layer and a skip connection structure, construct a multi-layer residual unit using different numbers of inverse residual blocks, construct a backbone network using multi-layer residual units, and construct a multi-scale detection network using an upsampling layer, a downsampling layer, an addition layer, a decoupling head and a non-maximum suppression layer; A second building module is configured to build a small target detection model using the brightness enhancement network, the backbone network and the multi-scale detection network; The enhancement module is configured to obtain a training image, input the training image into the small target detection model, and perform brightness enhancement on the training image through the brightness enhancement network to obtain an enhanced image; An extraction module, configured to extract multiple scale features of the enhanced image through residual units of different layers in the backbone network; a fusion module configured to fuse multiple scale features through the multi-scale detection network and determine a training detection result based on the fusion result; The optimization module is configured to calculate the loss based on the training image, the enhanced image, the training detection result and the label of the training image, and optimize the model parameters of the small target detection model based on the loss to complete the training of the small target detection model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.