A low-light target detection method based on pyramid enhancement network

Abstract

The present invention relates to the field of visual perception technology, and discloses a method for target detection under low light conditions based on a pyramid enhancement network. A pyramid enhancement network is constructed to enhance images and capture potential information in the images. The pyramid enhancement network first decomposes the image into multiple components of different resolutions through a Laplacian pyramid. In the components of each scale, a detail processing module and a low-frequency enhancement filter are constructed to enhance the components. The detail processing module consists of a context branch and an edge branch. The context branch globally enhances the components by capturing long-range dependencies, and the edge branch enhances the texture of the components. The low-frequency enhancement filter obtains low-frequency semantic information and blocks high-frequency noise through a dynamic low-pass filter to enrich feature information. After enhancing the low-light image, the pyramid enhancement network provides more information for the subsequent target detection model, thereby improving the performance of the detector.

Description

Technical Field

[0001] The present invention relates to the field of visual perception technology, and in particular to a method for detecting targets in low light conditions based on a pyramid enhancement network. Background Art

[0002] In recent years, the emergence of convolutional neural networks (CNNs) has fueled the development of object detection. Numerous detectors have been proposed, each with increasingly improved performance. Many have also been deployed in real-world scenarios, such as autonomous driving. However, most existing detectors are derived from high-quality images captured under normal conditions. Real-world environments often contain many adverse lighting conditions, such as nighttime, low light levels, and overexposure. This degrades image quality and impacts detector performance, making it impossible to provide reliable predictions under these conditions. Visual perception models enable unmanned systems to understand the environment, laying the foundation for subsequent tasks such as trajectory planning. This requires robust object detection or semantic segmentation models. If images are appropriately enhanced based on environmental conditions, restoring more potential information about originally blurred objects, object detection models can adapt to diverse low-light environments. This presents a significant challenge in the practical application of these models.

[0003] Numerous methods have been proposed to address the robustness of nighttime scenes. Numerous low-light enhancement models have been proposed to restore image detail and mitigate the effects of poor lighting conditions. However, these models overly focus on enhancing the human visual experience, increasing brightness while also amplifying noise. Their complex structures hinder the real-time performance of the detector after image enhancement. Most of these methods cannot be trained end-to-end with the detector and require supervised learning using paired low-light and normal images. Low-light object detection can also be considered a domain adaptation problem. Some existing techniques use adversarial learning to transfer models from normal to low-light conditions. However, these techniques focus on matching data distributions and ignore the potential information contained in low-light images. In the past two years, some existing techniques have proposed differentiable image processing (DIP) modules for image enhancement, which are trained end-to-end with the detector. DIP uses a CNN to predict filter parameters based on image brightness, color, and other parameters, adaptively processing the image. However, DIP's filters are traditional methods, such as white balance, and their image enhancement effect is limited.

[0004] The current work on dark target detection has the following main limitations: (1) Images appear blurred in dark scenes, resulting in loss of details; (2) Although existing image enhancement models or filters can improve the visual effects of images, they also amplify noise. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a low-light target detection method based on a pyramid enhancement network, which enhances low-light images through a pyramid enhancement network and captures potential features therein, thereby improving the performance of the target detection model in the dark.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A low-light target detection method based on a pyramid enhancement network is used to detect targets in images of night scenes, including the following steps:

[0008] Step 1: Decompose the image I1 into components of different resolutions through the Laplacian pyramid;

[0009] Step 2: Construct a detail processing module to perform detail enhancement on the components of different scales output by the Laplacian pyramid. Specifically, this includes global enhancement of the components through the context branch and texture enhancement of the components through the edge branch. The globally enhanced components and the components processed by the edge branch are tensor-concatenated and the detail processing results are obtained through a convolutional layer.

[0010] Step 3: Construct a low-frequency enhancement filter to capture the low-frequency semantics of the component and block the high-frequency noise to obtain a low-frequency enhancement result;

[0011] Step 4: Concatenate the detail processing results and the low-frequency enhancement results into tensors and fuse the features through a convolution layer to obtain the convolution-enhanced component.

[0012] Step 5: Perform Laplacian pyramid reconstruction on the components after convolution enhancement to obtain image H1 with the same resolution as image I1;

[0013] Step 6: Use image H1 as input to the target detector to obtain the detection result.

[0014] Furthermore, step one specifically includes:

[0015] Step 1A: Acquire an image I1 having channels R, G, and B through a camera;

[0016] Step 1B: Decompose the image I1 into components of different resolutions using the Laplacian pyramid, specifically including:

[0017] Step 1 B1: Obtain sub-images of different resolutions of image I1 through Gaussian pyramid. Each operation of Gaussian pyramid includes Gaussian filtering and downsampling:

[0018] G(I1)=Down(Gaussian(I1));

[0019] Where G(·) represents the Gaussian pyramid operation, Down(·) represents downsampling, and Gaussian(·) represents the Gaussian filter;

[0020] Step 1 B2: The information lost during the Gaussian pyramid operation constitutes the components of the Laplacian pyramid. The Laplacian pyramid is defined as:

[0021] L i =G i -Up(G i+1 );

[0022] Among them L i is the i-th level component of the Laplacian pyramid, G i Represents the i-th layer of the Laplacian pyramid, i = 1, 2, 3, ..., and Up represents the upsampling operation.

[0023] Furthermore, step 2 specifically includes:

[0024] Step 2A: When globally enhancing the components through the context branch, first process the output component L of the Laplacian pyramid through the residual block to obtain the long-range dependency L″ of the feature:

[0025] L″=L+γ(F1(L′));

[0026] The intermediate variable L′=σ(F2(L))·L, where F1 and F2 are both convolutional layers, γ is the Leaky ReLU activation function, and σ is the Softmax function. L″ is then processed by a residual block to obtain the globally enhanced component L CB ;

[0027] Step 2B: Enhance the texture of the component through the edge branch. First, the output component L of the Laplacian pyramid is processed by the horizontal Sobel operator and the vertical Sobel operator respectively and then added. Then, the edge is extracted again through a convolution layer and the residual operation is performed. The component after edge branch processing is recorded as L EB :

[0028] L EB =F3(Sobel h (L)+Sobel w (L))+L;

[0029] Among them Sobel h 、Sobel w They are the horizontal Sobel operator and the vertical Sobel operator respectively, and F3 is the convolution layer;

[0030] Step 2C: L CB and L EBAfter tensor splicing, a convolution layer is passed to obtain the result of detail processing L DPM .

[0031] Furthermore, step three specifically includes:

[0032] Step 3A: Perform initial feature extraction on different components of the Plaus pyramid output through a convolutional layer to obtain preliminary features;

[0033] Step 3B: Use a dynamic low-pass filter constructed by adaptive average pooling to capture the low-frequency information of each preliminary feature, then restore it to its original size through interpolation upsampling and perform tensor splicing;

[0034] Step 3C: Use a convolutional layer to extract features from the concatenated components of the tensors in step 3B to obtain the low-frequency enhancement result L LEM .

[0035] Furthermore, when performing Laplacian pyramid reconstruction on the convolution enhanced components in step 5:

[0036]

[0037] Among them, H i represents the reconstruction output of the i-th scale, Represents the component of the i-th scale after convolution enhancement, Up represents upsampling, G i+1 Represents the i+1th level of the Laplacian pyramid.

[0038] Compared with the prior art, the beneficial technical effects of the present invention are:

[0039] This paper uses an enhancement network based on a Laplacian pyramid to process low-light images and output an enhanced image. Furthermore, unlike typical low-light enhancement models that require paired normal-low-light images, this method can be directly trained end-to-end with an object detector and only requires a standard detection loss.

[0040] The method of the present invention decomposes the low-light image into components through a Laplacian pyramid, and recovers the potential information in the low-light image through detail processing and low-frequency information capture.

[0041] The method of the present invention combines a pyramid enhancement network with an object detector to construct an end-to-end dark object detection model that is robust to scenes with different degrees of low light. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to illustrate the present invention more clearly, the following will briefly introduce the drawings required for use in the present invention. Obviously, the drawings described below are some embodiments of the present invention. For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without paying any creative work.

[0043] Figure 1 A schematic diagram of a pyramid enhancement network provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of detailed processing of components after decomposition of a Laplacian pyramid provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of performing low-frequency enhancement filter processing on components after Laplacian pyramid decomposition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. It should be understood that the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0047] Second, “one embodiment” or “embodiment” referred to in the present invention refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention.

[0048] Before introducing the embodiments of the present invention, the following definitions are given for the relevant terms involved:

[0049] Laplacian Pyramid: The Laplacian Pyramid is the difference between a Gaussian Pyramid and the upsampled image above it. Upsampling here is generally performed using interpolation. The Laplacian Pyramid is actually created to achieve image reconstruction using the Gaussian Pyramid. It is generated based on the Gaussian Pyramid.

[0050] Sobel operator: Sobel is a discrete differential operator that combines Gaussian smoothing and differential derivative operations. This operator can use local differences to solve the edges of the image.

[0051] The work of the present invention is related to target detection at night. The present invention constructs a pyramid enhancement network to enhance the image and capture the potential information in the image. The pyramid enhancement network first decomposes the image into multiple components of different resolutions through the Laplacian pyramid. In the components of each scale, a detail processing module (DPM) and a low-frequency enhancement filter (LEF) are constructed to enhance the components. The DPM consists of a context branch and an edge branch. The context branch globally enhances the components by capturing long-range dependencies, and the edge branch enhances the texture of the components. LEF obtains low-frequency semantic information and blocks high-frequency noise through a dynamic low-pass filter to enrich feature information. The pyramid enhancement network and the target detector are trained end-to-end to construct a dark target detection model. Only a general detection loss function is used during model training. After enhancing the low-light image, the pyramid enhancement network provides more information for the subsequent target detection model, thereby improving the performance of the detector.

[0052] The dark object detection model in the present invention includes five modules: Laplacian pyramid decomposition, detail processing module, low-frequency enhancement filter, Laplacian pyramid reconstruction, and object detector, among which:

[0053] Laplacian pyramid decomposition: construct a Laplacian pyramid through the image to decompose the image into components of different resolutions;

[0054] Detail processing module: Global enhancement of components is performed through context branches and texture enhancement of components is performed through edge branches, and finally tensor splicing and fusion features are performed;

[0055] Low-frequency enhancement filter: Features are filtered through adaptive average pooling, and a dynamic low-pass filter is constructed through pooling at multiple scales, which allows low-frequency semantics to pass through while blocking high-frequency noise.

[0056] Laplacian pyramid reconstruction: The enhanced components are reconstructed into the original high-resolution image through the Laplacian pyramid.

[0057] Object detector: The object detector used has the same network structure as the existing technology, and directly forms an end-to-end framework with the pyramid enhancement network.

[0058] Below, we will refer to Figure 1 Describes a method for low-light object detection based on pyramid enhancement networks. Figure 1 In , Input means input, image means image, Output means output, Concat means concatenation, and Conv means convolution layer; Figure 1 As shown, the specific implementation steps of this embodiment are as follows:

[0059] Step 1: Decompose the image I1 into components of different resolutions (i.e., different scales) using a Laplacian pyramid. This specifically includes the following steps:

[0060] Step 1A: Acquire an image I1 having channels R, G, and B through a camera;

[0061] Step 1B: Decompose the image I1 into four components of different resolutions using an image pyramid, including:

[0062] Step 1 B1: Obtain sub-images of different resolutions through the Gaussian pyramid. Each operation of the Gaussian pyramid is Gaussian filtering and downsampling, which can be expressed as:

[0063] G(I1)=Down(Gaussian(I1));

[0064] Where G(I1) represents the Gaussian pyramid operation function, Down represents downsampling, Gaussian represents the Gaussian filter, and the Gaussian kernel size is 5 × 5. After each Gaussian pyramid operation, the width and height of the image are halved, and the resolution becomes 1 / 4 of the original.

[0065] Step 1 B2: The information lost in the Gaussian pyramid process constitutes the components of the Laplacian pyramid; the Laplacian pyramid is defined as:

[0066] L i =G i -Up(G i+1 );

[0067] Among them L i is the i-th level component of the Laplace pyramid, G i Represents the i-th layer of the Laplacian pyramid, i = 1, 2, 3, ..., and Up represents the upsampling operation.

[0068] Step 2: Construct a detail processing module (DPM) to perform detail enhancement on the components of different scales output by the Laplacian pyramid. Specifically, this includes global enhancement of the components through the context branch and texture enhancement of the components through the edge branch. The globally enhanced components and the components processed by the edge branch are tensor-concatenated and the detail processing results are obtained through the convolution layer. Specifically, this includes:

[0069] Step 2A: If Figure 2 As shown in Figure 1, when globally enhancing the components through the context branch, the output component L of the Laplacian pyramid is first processed by the residual block, and the feature channel is increased from 3 to 32, and the long-range dependency L″ of the feature is obtained:

[0070] L″=L+γ(F1(L′));

[0071] The intermediate variable L′=σ(F2(L))·L, where F1 and F2 are both convolutional layers with a kernel size of 3, γ is the Leaky ReLU activation function, and σ is the Softmax function; L″ is then processed by a residual block, and the channel is reduced from 32 to 3, obtaining the globally enhanced component L CB .

[0072] Step 2B: If Figure 2 As shown in the figure, the texture of the component is enhanced by the edge branch. First, the component L of the output of the Laplacian pyramid is processed by the horizontal Sobel operator and the vertical Sobel operator respectively and then added. Then, the edge is extracted again through a convolution layer. In order to promote the flow of features, a residual operation is performed; the component after edge branch processing is recorded as L eB :

[0073] L EB =F3(Sobel h (L)+Sobel w (L))+L;

[0074] Among them Sobel h 、Sobel w They are the horizontal Sobel operator and the vertical Sobel operator respectively, and F3 is another set of convolutional layers with a kernel size of 3;

[0075] Step 2C: L CB and L EB After tensor splicing, the detail processing result L is obtained through a convolution layer with a kernel size of 3 DPM .

[0076] Figure 2 In the figure, EB represents edge branch, CB represents context branch, h represents horizontal, w represents vertical, and Res Block represents residual block.

[0077] Step 3: Construct a low-frequency enhancement filter (LEF) to capture the semantics of low frequencies in the component and block high-frequency noise to obtain low-frequency enhancement results; specifically, it includes:

[0078] Step 3A: Perform initial feature extraction on the different components of the Plaus pyramid output through a convolutional layer with a kernel size of 1x1 to obtain preliminary features;

[0079] Step 3B: If Figure 3 As shown in the figure, a dynamic low-pass filter is constructed by adaptive average pooling with kernel sizes of 1, 2, 3, and 6 to capture the low-frequency information of each preliminary feature, and then restored to the original size by interpolation upsampling, which are denoted as f1, f2, f3, and f4 respectively. Finally, f1, f2, f3, and f4 are tensor spliced;

[0080] Step 3C: Use a convolutional layer with a kernel size of 3x3 to extract features from the concatenated components of the tensors in step 3B to obtain the low-frequency enhancement result L LEM .

[0081] Step 4: Process the details DPM And the low frequency enhancement result L LEM , perform tensor splicing and fuse features through a convolution layer to obtain the convolution-enhanced component L enhance ;

[0082] Step 5: Perform Laplacian pyramid reconstruction on the components after convolution enhancement to obtain image H1 with the same resolution as image I1; specifically: through the component L at each scale enhance The image is reconstructed and the process is described as:

[0083]

[0084] Among them, H i Represents the reconstructed output, and finally the enhanced image I is obtained enhabce , which is the final image H1.

[0085] Step 6: Use image H1 as input to the object detector to obtain detection results.

[0086] After enhancing the image, predictions are made using an object detector. The model used here is YOLOv3, which, combined with a pyramid enhancement network, forms an end-to-end trained dark detection framework. This simplifies the training process by requiring only a standard detection loss. The final detection results are the x, y, w, h, and confidence values ​​of multiple predicted boxes, representing the horizontal and vertical coordinates of the top left corner of the predicted box, the width and height of the rectangle, and the confidence level.

[0087] It should be noted that the object detector can be any convolutional neural network capable of detecting objects, which can be pre-trained or trained on local data. Supervised learning is performed using the rectangles of objects that have been annotated in the image.

[0088] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

Claims

1. A low-light target detection method based on a pyramid enhancement network is used to detect targets in images of night scenes, comprising the following steps: Step 1: Transform the image through the Laplacian pyramid Decompose into components of different resolutions; Step 2: Construct a detail processing module to perform detail enhancement on the components of different scales output by the Laplacian pyramid, specifically including global enhancement of the components through the context branch and texture enhancement of the components through the edge branch; The components after global enhancement and edge branch processing are concatenated into tensors, and the detail processing results are obtained through the convolution layer; specifically, the following steps are included: Step 2A: When globally enhancing the components through the context branch, first process the component L of the Laplacian pyramid output through the residual block to obtain the long-range dependency of the features. : ; Intermediate variables , 、 Both are convolutional layers. is the Leaky ReLU activation function, It is the Softmax function; then it is passed through a residual block Processing is performed to obtain the globally enhanced component ; Step 2B: Texture enhancement of components is performed through edge branches. First, the components L of the Laplacian pyramid output are processed by the horizontal Sobel operator and the vertical Sobel operator respectively and then added. Then, the edges are extracted again through a convolution layer and the residual operation is performed. The components after edge branch processing are recorded as : ;in 、 They are the horizontal Sobel operator and the vertical Sobel operator, is the convolutional layer; Step 2C: and After tensor splicing, a convolution layer is passed to obtain the result of detail processing ; Step 3: Construct a low-frequency enhancement filter to capture the low-frequency semantics of the component and block the high-frequency noise to obtain a low-frequency enhancement result; Step 4: Concatenate the detail processing results and the low-frequency enhancement results into tensors and fuse the features through a convolution layer to obtain the convolution-enhanced component. Step 5: Perform Laplace pyramid reconstruction on the components after convolution enhancement to obtain Images of the same resolution ; Step 6: Image As input to the target detector, the detection result is obtained.

2. The method for low-light target detection based on a pyramid enhancement network according to claim 1, wherein: Step 1 specifically includes: Step 1A: Acquire an image with channels R, G, and B through the camera ; Step 1B: Transform the image through the Laplacian pyramid Decomposed into components of different resolutions, including: Step 1 B1: Obtain image through Gaussian pyramid For sub-images of different resolutions, each operation of the Gaussian pyramid includes Gaussian filtering and downsampling: ； in represents the Gaussian pyramid operation, represents downsampling, represents a Gaussian filter; Step 1 B2: The information lost during the Gaussian pyramid operation constitutes the components of the Laplacian pyramid. The Laplacian pyramid is defined as: ； in is the i-th level component of the Laplacian pyramid, Represents the i-th layer of the Laplacian pyramid, i=1,2,3..., and Up represents the upsampling operation.

3. The method for low-light target detection based on a pyramid enhancement network according to claim 1, wherein: Step three specifically includes: Step 3A: Perform initial feature extraction on different components of the Plaus pyramid output through a convolutional layer to obtain preliminary features; Step 3B: Use a dynamic low-pass filter constructed by adaptive average pooling to capture the low-frequency information of each preliminary feature, then restore it to its original size through interpolation upsampling and perform tensor splicing; Step 3C: Use a convolutional layer to extract features from the concatenated components of the tensors in step 3B to obtain low-frequency enhancement results. .

4. The method for low-light target detection based on a pyramid enhancement network according to claim 1, wherein: In step 5, when reconstructing the Laplacian pyramid of the convolution-enhanced components: in, represents the reconstruction output of the i-th scale, Represents the component of the i-th scale after convolution enhancement, represents upsampling, Represents the i+1th level of the Laplacian pyramid.

Images