Low-light human pose estimation method based on image frequency feature enhancement

Abstract

The application belongs to the technical field of human posture estimation, and discloses a very low light human posture estimation method based on image frequency feature enhancement, which comprises the following steps: collecting very low light human images to construct a training data set; constructing a very low light human posture estimation model based on image frequency feature enhancement; in the training stage, the frequency-aware image enhancement module is used to enhance the training data, and then the enhanced data is input into the network trunk for training to obtain a trained model; in the testing stage, the testing data set is input into the trained network model to obtain human posture estimation results. The network model provided by the application can use the frequency-aware image enhancement module to guide the network to generate more abundant and clearer image features, realize the collaborative optimization of image enhancement and human posture estimation, and thus improve the very low light human posture estimation accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of human posture estimation, and in particular relates to an extremely low-light human posture estimation method based on image frequency division feature enhancement. Background Art

[0002] As one of the basic tasks of computer vision, human pose estimation aims to locate key points of the human body to obtain human pose information. It is widely used in fields such as action recognition, human-computer interaction, and intelligent monitoring. In recent years, human pose estimation methods based on deep learning have made significant progress on multiple benchmark datasets. However, most existing research focuses on natural lighting environments, and there is little in-depth discussion on the problem of human pose estimation in extremely low-light environments. In addition, the difference in image features in extremely low-light environments puts more stringent requirements on human pose estimation. Taking extremely low-light application scenarios such as nighttime autonomous driving and all-weather security monitoring as an example, their extremely low-light images generally have the characteristics of reduced clarity and detail expression capabilities. This further leads to the problems of structural confusion and unclear texture in the features extracted by existing human pose estimation models in extremely low-light environments, making it difficult to accurately identify human joints.

[0003] Although previous studies have attempted to improve the performance of human pose estimation in extreme low light conditions by fine-tuning pre-trained models on extreme low light datasets or applying existing image enhancement techniques, these methods have significant limitations. First, simple fine-tuning strategies may affect the performance of the model under natural light conditions, which violates the principle of model generalization. Second, most existing extreme low light image enhancement technologies are dedicated to meeting the needs of human visual perception rather than optimizing for the needs of human pose estimation tasks, which results in only suboptimal gains in improving the performance of extreme low light human pose estimation. In addition, some extreme low light image enhancement methods rely on strictly paired extreme low light and natural light images, but such paired data are often difficult to obtain in practical applications.

[0004] Therefore, there is an urgent need for a human pose estimation method for extremely low-light images, which can improve the quality of extremely low-light images without relying on natural light images, thereby improving the detection accuracy of extremely low-light human pose estimation. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides an extremely low-light human posture estimation method based on image frequency division feature enhancement. This method constructs an extremely low-light human posture estimation model based on image frequency division feature enhancement, which significantly improves the robustness and accuracy of the human posture estimation model under different lighting conditions.

[0006] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0007] The present invention is a method for extremely low-light human posture estimation based on image frequency division feature enhancement, which specifically comprises the following steps:

[0008] Step 1: Collect extremely low-light human images to build training and test datasets;

[0009] Step 2: Construct an extremely low-light human pose estimation network model based on image frequency division feature enhancement. The extremely low-light human pose estimation network model includes a frequency-aware image enhancement module and a backbone network. The frequency-domain-aware image enhancement module is used to enhance the training data set constructed in step 1. The enhanced data is then input into the backbone network for training to obtain a trained extremely low-light human pose estimation network model.

[0010] Among them, the frequency-aware image enhancement module includes a Laplace decomposition layer, a texture modulation module, a structure enhancement module and a spatial alignment reconstruction layer. The frequency-domain-aware image enhancement module is used to decouple the high-frequency feature components and low-frequency feature components of the extremely low-light image, and enhance the global structural information in the low-frequency feature components and the texture edge information in the high-frequency feature components respectively. The effective fusion and structural consistency of the high-frequency feature components and the low-frequency feature components are ensured through spatial alignment reconstruction to obtain an enhanced extremely low-light image. The backbone network is used to extract multi-scale features of the enhanced extremely low-light image and predict the final human posture estimation result;

[0011] Step 3: Input the test data set into the trained network model obtained in step 2 to obtain the human posture estimation result.

[0012] A further improvement of the present invention is that: Step 1 of collecting extremely low-light human images to construct a training data set is specifically as follows: extremely low-light human images are collected from the low-light human image data set ExLPose to construct a training data set.

[0013] A further improvement of the present invention is that in step 2, constructing an extremely low-light human posture estimation network model based on image frequency division feature enhancement specifically includes the following steps:

[0014] Step 2.1, extracting image features of different frequencies of the extreme low-light image and constructing a Laplace decomposition layer. The Laplace decomposition layer performs multi-scale frequency decomposition on the extracted extreme low-light image through a Laplace pyramid, decoupling the extreme low-light image features into high-frequency feature components and low-frequency feature components;

[0015] Step 2.2: Construct a texture modulation module and a structure enhancement module. The texture modulation module adaptively learns the spatial weights of the fusion of high-frequency feature components at different scales, achieving complementarity and enhancement between high-frequency feature components to obtain enhanced high-frequency image features. The structure enhancement module consists of two parts: channel frequency domain modulation and spatial frequency domain modulation. They respectively model the dependencies between spatial frequency domain information and channel frequency domain information to obtain enhanced low-frequency image features to better capture the global structural information of the low-frequency components of the image.

[0016] Step 2.3: Construct a spatial alignment reconstruction layer to fuse the enhanced high-frequency image features with the low-frequency image features based on the spatial alignment strategy to obtain an enhanced extreme low-light image.

[0017] Step 2.4: Build the backbone network, namely the HRNet network model, extract the multi-scale features of the extremely low-light image enhanced in step 2.3, classify the joint points and predict their positions, and predict the final human posture estimation result.

[0018] A further improvement of the present invention is that when the frequency-domain-aware image enhancement module is used to enhance the training data, the Laplace decomposition layer performs the following steps:

[0019] Step 2.1.1. Use the Gaussian pyramid to construct a hierarchical structure for the extreme low-light image. The input extreme low-light image with a resolution of h×w is decomposed into a low-frequency feature component and multiple high-frequency feature components:

[0020]

[0021] Among them, G n (I) represents the extremely low-light image at the nth level of the Gaussian pyramid, N represents the total number of pyramid levels, and B n (G n-1 (I)) represents the extremely low light image G of the previous layer n-1 (I) Perform 2D Gaussian filtering, where D(·) represents a 2x downsampling operation;

[0022] Step 2.1.2: Repeated Gaussian filtering and downsampling filter out the high-frequency feature components and define the image at the highest level of the Gaussian pyramid as the low-frequency feature component LF. N :

[0023] LF N =G N (I)

[0024] Step 2.1.3: The high-frequency characteristic component HF decoupled from each layer of the Laplace pyramid n Obtained by subtracting two adjacent layers of Gaussian pyramid images:

[0025] HF n =Gn (I)-U(G n+1 (I))

[0026] Where 1≤n≤N-1, U(.) represents a 2x upsampling operation.

[0027] The present invention is further improved in that: the texture modulation module includes high-frequency feature scale unification, i.e., Resize and adaptive spatial weight learning and weighted fusion, that is, unifying high-frequency feature components of different scales to the same size, and for the high-frequency feature component HF of the n-th layer branch n When the frequency domain-aware image enhancement module is used to enhance the training data, the texture modulation module performs the following steps:

[0028] Step 2.2.1, the texture modulation module first samples the high-frequency feature components of other branches to ensure that they are consistent with the size of the high-frequency feature components of the nth layer, and defines Represents the pixel value of the high-frequency component at position (i, j) sampled from the n1th layer to the n2th layer;

[0029] Step 2.2.2: Weight and fuse the different high-frequency feature components that are unified to the same size. Utilizing the fact that the texture information of different high-frequency feature components has correlation in spatial distribution, the fusion strategy designed by the present invention is expressed as follows:

[0030]

[0031] in, Represents the pixel value at position (i, j) in the high-frequency feature component after fusion of the nth layer, Refers to the spatial importance weights of the feature maps of the three high-frequency feature components at different levels in the nth layer. The corresponding weight map is obtained by using 1×1 convolution, and the sum of the spatial importance weights of the feature maps of the three high-frequency feature components at different levels is constrained to be 1:

[0032]

[0033] Through the fusion strategy, high-frequency feature components of all levels are adaptively fused at their respective scales.

[0034] A further improvement of the present invention is that the structure enhancement module includes channel frequency domain modulation based on energy compression and spatial frequency domain modulation based on parameter weighting. When the frequency domain-aware image enhancement module is used to enhance the training data in step 3, the structure enhancement module performs the following steps:

[0035] Step 2.3.1: Input low-frequency feature component LF NThrough 3D attention mechanism encoding, the low-frequency feature component LF of the input is re-weighted by optimizing the energy function N , for the low-frequency characteristic component LF N The feature map F∈R of the processed input C ×H×W , calculate the mean according to the channel and variance Then calculate the energy value of each pixel by minimizing the energy function in the formula:

[0036]

[0037] Among them, λ is a hyperparameter, e ij Represents the target pixel t ij Energy value, i∈{0, 1, ..., H-1}, j∈{0, 1, ..., W-1}, pixel weight

[0038]

[0039] Step 2.3.2, re-weight the points Adjust to weight

[0040]

[0041] in, represents the sum of the weight values ​​of each channel, and g(·) represents the activation function;

[0042] Step 2.3.3. Weight The spectrum graph f is further transformed into the frequency domain through discrete cosine transform (DCT) uv :

[0043]

[0044] f c =zigzag(f uv )

[0045] Among them, f uv Represents the value of the discrete cosine transformed spectrum f at the position (u, v), u∈{0,1,…,H-1}, v∈{0,1,…,W-1}, i, j refer to the position, f c is the frequency domain vector;

[0046] Step 2.3.4, use zig-zag compression to convert the spectrum f uv Further compressed into a compact frequency domain vector f c ∈R C×Dand with a learnable filter Filter∈R C×H×W Perform element-wise multiplication to improve representation capability. The expression is as follows:

[0047] f c1 =f c FC(R(Filter))

[0048] Among them, R C×D The filter parameters are changed from R to C×H×W The resize operation includes the reshape operation R(·) and the linear layer FC(·), D < H × W, D is the dimension after H × W is compressed by zig-zag, and the channel weight f is obtained by linear layer compression. c2 ∈R C , and weighted fused with the feature map F to obtain the channel frequency domain modulation enhanced feature F ′ :

[0049] F ′ =FC(f c1 )×F

[0050] Among them, F ′ ∈R C×H×W , FC(·) represents a linear layer, and × represents a multiplication operation along the channel;

[0051] Step 2.3.5: Enhance the channel frequency domain modulation feature F ′ The discrete cosine transform (DCT) is used to transform the feature into the frequency domain and the learnable filter is used to capture the weight relationship between the frequency bands. The information of different frequency bands in the frequency domain features is weighted and fused. That is, the input feature map F is transformed into ′ ∈R C×H×W Converted into frequency domain feature map f∈R C ×H×W , using learnable filters to capture the importance relationship between frequency bands, indirectly making more flexible interactions between different spatial locations of the image, and obtaining frequency domain information f * :

[0052] f * =f·Filter

[0053] Where Filter∈R C×H×W Represents the parameters of the learnable filter and is learned by the network;

[0054] Step 2.3.6: frequency domain information f * The inverse conversion is performed into spatial information for subsequent reconstruction. The feature map of a certain channel is processed. The spatial frequency domain modulation expression is as follows:

[0055]

[0056] in F″ ij The value at position (i, j) of a channel feature map after inverse discrete cosine transform (IDCT) is the enhanced feature after spatial frequency domain modulation.

[0057] A further improvement of the present invention is that the spatial alignment reconstruction layer is implemented by a spatial alignment mechanism based on deformable convolution. When the frequency-domain-aware image enhancement module is used to enhance the training data in step 3, the spatial alignment reconstruction layer performs the following steps:

[0058] Step 2.4.1, adjust the low-frequency feature component according to the spatial position information provided by the high-frequency feature component, define the spatial position information adjustment amount as an offset, and represent it with a 2D feature map, where each offset value is regarded as a low-frequency feature component LF n Each point after upsampling is related to its HF n-1 The moving distance between the corresponding points in 2D space, the low-frequency feature component fusion reconstruction process expression is as follows:

[0059]

[0060] in, is the high-frequency feature component of the n-1th layer after spatial alignment reconstruction, f o (·),f a (·) denotes the offset Δ learned from the spatial difference n and a function to align features with offsets, f a (·), f o (·) implemented using deformable convolution,

[0061] Step 2.4.2: For the fusion reconstruction of different high-frequency components, the following method is adopted:

[0062]

[0063] Where n=1,2,3,4.

[0064] A further improvement of the present invention is that: in step 2.4, the backbone network is any CNN-based network, and the predicted joint point positions are determined by the heat map finally output by the backbone network.

[0065] A further improvement of the present invention is that the loss function for training the extremely low-light human pose estimation network model uses the MSE loss function, which is expressed as follows:

[0066]

[0067] Among them, N represents the number of samples, W represents the number of types of human joints, Represents the predicted heat map of the wth human key point of the nth sample in a batch, Represents the heat map of the wth human key point label of the nth sample in a batch.

[0068] The beneficial effects of the present invention are:

[0069] The method of the present invention proposes an image enhancement paradigm that shifts the problem framework from human visual perception to machine perception, providing a directly relevant information enhancement strategy for the task of human pose estimation in extremely low-light environments, breaking away from the limitations of strictly paired natural light images, and achieving collaborative optimization of image enhancement and human pose estimation.

[0070] Based on the characteristic that different frequency components of an image have different physical meanings, the method of the present invention extracts and enhances task-related robust features from a frequency perspective, designs an image enhancement module based on frequency perception, and seamlessly integrates it into the front end of the human pose estimation network to construct an extremely low-light human pose estimation model based on image frequency division feature enhancement. This avoids the tedious alignment processing of traditional domain adaptive methods, enables the model to maintain high computational efficiency under various lighting conditions, and significantly improves the robustness and accuracy of the human pose estimation model under different lighting conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Schematic diagram of the workflow of the extremely low-light human posture estimation method of the present invention.

[0072] Figure 2 This is a network structure diagram of the image enhancement module of the present invention.

[0073] Figure 3 This is a network structure diagram of the texture modulation module in the image enhancement module of the present invention.

[0074] Figure 4 This is a network structure diagram of the structure enhancement module in the image enhancement module of the present invention.

[0075] Figure 5 This is a network structure diagram of the spatial alignment reconstruction layer in the image enhancement module of the present invention.

[0076] Figure 6 Schematic diagram of the workflow of the training phase of the present invention.

[0077] Figure 7 Schematic diagram of the working process of the testing phase of the present invention.

[0078] Figure 8 A comparison chart for visualization based on different lighting conditions.

[0079] Figure 9 This is a comparison chart of the low-light image prediction results under different lighting conditions. DETAILED DESCRIPTION

[0080] The following diagrams illustrate embodiments of the present invention. For clarity, many practical details are included in the following description. However, it should be understood that these practical details are not intended to limit the present invention. In other words, in some embodiments of the present invention, these practical details are not essential.

[0081] like Figure 1 As shown, the present invention provides an extremely low-light human pose estimation method based on image frequency division feature enhancement. An image enhancement module is added to the original backbone network, thereby making it easier to perform human pose estimation in aurora scenes and improving the accuracy of human pose estimation. The method specifically includes the following steps:

[0082] Step 1: Collect extremely low-light human images to construct a training dataset. Specifically, collect extremely low-light human images from the low-light human image dataset ExLPose to construct a training dataset and a test dataset, where the ratio of the training dataset to the test dataset is 4:1.

[0083] Step 2: Construct an extremely low-light human pose estimation network model based on image frequency division feature enhancement. The extremely low-light human pose estimation network model includes a frequency-aware image enhancement module and a backbone network. The frequency-domain-aware image enhancement module is used to enhance the training data set constructed in step 1. The enhanced data is then input into the backbone network for training to obtain a trained extremely low-light human pose estimation network model.

[0084] Among them, the frequency-aware image enhancement module includes a Laplace decomposition layer, a texture modulation module, a structure enhancement module and a spatial alignment reconstruction layer. The frequency-domain-aware image enhancement module is used to decouple the high-frequency feature components and low-frequency feature components of the extremely low-light image, and enhance the global structural information in the low-frequency feature components and the texture edge information of the high-frequency feature components respectively. Through spatial alignment reconstruction, the effective fusion and structural consistency of the high-frequency feature components and the low-frequency feature components are ensured to obtain an enhanced extremely low-light image. The backbone network is used to extract the multi-scale features of the enhanced extremely low-light image and predict the final human posture estimation result.

[0085] like Figure 2 As shown in the figure, the construction of an extremely low-light human pose estimation network model based on image frequency division feature enhancement specifically includes the following steps:

[0086] Step 2.1: Extract image features at different frequencies from the extreme low-light image and construct a Laplacian decomposition layer for subsequent feature enhancement and fusion. The Laplacian decomposition layer performs multi-scale frequency decomposition on the extracted extreme low-light image using a Laplacian pyramid, decoupling the extreme low-light image features into high-frequency and low-frequency feature components.

[0087] When the frequency-domain-aware image enhancement module is used to enhance the training data, the Laplace decomposition layer performs the following steps:

[0088] Step 2.1.1. Use the Gaussian pyramid to construct a hierarchical structure for the extreme low-light image. The input extreme low-light image with a resolution of h×w is decomposed into a low-frequency feature component and multiple high-frequency feature components:

[0089]

[0090] Among them, G n (I) represents the extremely low-light image at the nth level of the Gaussian pyramid, N represents the total number of pyramid levels, and B n (G n-1 (I)) represents the extremely low light image G of the previous layer n-1 (I) Perform 2D Gaussian filtering, where D(·) represents a 2x downsampling operation;

[0091] Step 2.1.2: Since repeated Gaussian filtering and downsampling can filter out high-frequency feature components, the image at the highest level of the Gaussian pyramid can be defined as the low-frequency feature component LF. N :

[0092] LF N =G N (I)

[0093] Step 2.1.3: The high-frequency characteristic component HF decoupled from each layer of the Laplace pyramid n Obtained by subtracting two adjacent layers of Gaussian pyramid images:

[0094] HF n =G n (I)-U(G n+1 (I))

[0095] Where 1≤n≤N-1, U(.) represents a 2x upsampling operation.

[0096] Step 2.2, construct a texture modulation module and a structure enhancement module: used to enhance the image features of different frequencies of the extremely low-light image respectively, so as to reconstruct image features that are more conducive to the extremely low-light human posture estimation network model to perform human posture estimation; the texture modulation module adaptively learns the spatial weights of the fusion of high-frequency feature components of different scales, realizes the complementarity and enhancement between high-frequency feature components, and obtains enhanced high-frequency image features. The structure enhancement module consists of two parts: channel frequency domain modulation and spatial frequency domain modulation, which respectively model the dependencies between spatial frequency domain information and channel frequency domain information to obtain enhanced low-frequency image features, so as to better capture the global structural information of the low-frequency components of the image.

[0097] like Figure 3 As shown, the texture modulation module includes high-frequency feature scale unification, i.e., resizing, and adaptive spatial weight learning and weighted fusion, that is, unifying high-frequency feature components of different scales to the same size. When the frequency-domain-aware image enhancement module is used to enhance the training data, the texture modulation module performs the following steps:

[0098] Step 2.2.1: High-frequency characteristic component HF of the n-th layer branch n The texture modulation module first samples the high-frequency feature components of other branches to ensure that they are consistent with the size of the high-frequency feature components of the nth layer, and defines Represents the pixel value of the high-frequency component at position (i, j) sampled from the n1th layer to the n2th layer;

[0099] Step 2.2.2: Weight and fuse the different high-frequency feature components that are unified to the same size. Utilizing the fact that the texture information of different high-frequency feature components has correlation in spatial distribution, the fusion strategy designed by the present invention is expressed as follows:

[0100]

[0101] in, Represents the pixel value at position (i, j) in the high-frequency feature component after fusion of the nth layer, Refers to the spatial importance weights of the feature maps of the three high-frequency feature components at different levels in the nth layer. The corresponding weight map is obtained by using 1×1 convolution, and the sum of the spatial importance weights of the feature maps of the three high-frequency feature components at different levels is constrained to be 1:

[0102]

[0103] Through the fusion strategy, high-frequency feature components of all levels are adaptively fused at their respective scales.

[0104] like Figure 4As shown, the structure enhancement module includes channel frequency domain modulation based on energy compression and spatial frequency domain modulation based on parameter weighting. When the frequency domain-aware image enhancement module is used to enhance the training data in step 3, the structure enhancement module performs the following steps:

[0105] Step 2.3.1: Input low-frequency feature component LF N Through 3D attention mechanism encoding, the low-frequency feature component LF of the input is re-weighted by optimizing the energy function N , for the low-frequency characteristic component LF N The feature map F∈R of the processed input C ×H×W , calculate the mean according to the channel and variance Then calculate the energy value of each pixel by minimizing the energy function in the formula:

[0106]

[0107] Among them, λ is a hyperparameter, which is empirically set to 0.0001, e ij Represents the target pixel t ij Energy value, i∈{0, 1, ..., H-1}, j∈{0, 1, ..., W-1}, pixel weight

[0108]

[0109] Where F∈R C×H×W 3*32*24, H=32, W=24

[0110] Step 2.3.2: In order to prevent excessive weights from causing instability in training, use the idea of ​​normalization to re-adjust the weights of the points. Adjust to weight

[0111]

[0112] in, represents the sum of the weight values ​​of each channel, g(·) represents the activation function; the present invention adopts the LeakyReLU activation function.

[0113] Step 2.3.3. Weight The spectrum graph f is further transformed into the frequency domain through discrete cosine transform (DCT) uv :

[0114]

[0115] f c =zigzag(fuv )

[0116] Among them, f uv Represents the value of the discrete cosine transformed spectrum f at the position (u, v), u∈{0,1,…,H-1}, v∈{0,1,…,W-1}, i, j refer to the position, f c is the frequency domain vector;

[0117] Step 2.3.4: Then, refer to the JPEG standard and use the zig-zag compression method to convert the spectrum image f uv Further compressed into a compact frequency domain vector f c ∈R C×D and with a learnable filter Filter∈R C×H×W Perform element-wise multiplication to improve representation capability. The expression is as follows:

[0118] f c1 =f c FC(R(Filter))

[0119] Figure 3 In the example, the Resize operation includes the reshape operation R(·) and the linear layer FC(·), which changes the filter parameters from R C ×H×W Convert R C×D , where D<H×W, D is the dimension of H×W after zig-zag compression, and the present invention takes D=16. The channel weight f is obtained by linear layer compression. c2 ∈R C , and weighted fused with the feature map F to obtain the channel frequency domain modulation enhanced feature F ′ :

[0120] F ′ =FC(f c1 )×F

[0121] Among them, F ′ ∈R C×H×W , FC(·) represents a linear layer, and × represents a multiplication operation along the channel;

[0122] Step 2.3.5: Enhance the channel frequency domain modulation feature F ′ The discrete cosine transform (DCT) is used to transform the feature into the frequency domain and the learnable filter is used to capture the weight relationship between the frequency bands. The information of different frequency bands in the frequency domain features is weighted and fused. That is, the input feature map F is transformed into ′ ∈R C×H×W Converted into frequency domain feature map f∈R C ×H×W, using learnable filters to capture the importance relationship between frequency bands, indirectly making more flexible interactions between different spatial locations of the image, and obtaining frequency domain information f * :

[0123] f * =f·Filter

[0124] Where Filter∈R C×H×W Represents the parameters of the learnable filter and is learned by the network;

[0125] Step 2.3.6: frequency domain information f * The inverse conversion is performed into spatial information for subsequent reconstruction. The feature map of a certain channel is processed. The spatial frequency domain modulation expression is as follows:

[0126]

[0127] in F″ ij The value at position (i, j) of a channel feature map after inverse discrete cosine transform (IDCT) is the enhanced feature after spatial frequency domain modulation.

[0128] Step 2.3: Construct a spatial alignment reconstruction layer, and fuse the enhanced high-frequency image features and low-frequency image features based on the spatial alignment strategy to obtain an enhanced extreme low-light image.

[0129] like Figure 5 As shown, the spatial alignment reconstruction layer is implemented by a spatial alignment mechanism based on deformable convolution. When the frequency-domain-aware image enhancement module is used to enhance the training data in step 3, the spatial alignment reconstruction layer performs the following steps:

[0130] Step 2.4.1, adjust the low-frequency feature component according to the spatial position information provided by the high-frequency feature component, define the spatial position information adjustment amount as an offset, and represent it with a 2D feature map, where each offset value is regarded as a low-frequency feature component LF n Each point after upsampling is related to its HF n-1 The moving distance between the corresponding points in 2D space, the low-frequency feature component fusion reconstruction process expression is as follows:

[0131]

[0132]

[0133] in, is the high-frequency feature component of the n-1th layer after spatial alignment reconstruction, f o (·),f a (·) denotes the offset Δ learned from the spatial differencen and a function to align features with offsets, f a (·), f o (·) implemented using deformable convolution,

[0134] Step 2.4.2: For the fusion reconstruction of different high-frequency components, the following method is adopted:

[0135]

[0136] Where n = 1, 2, 3, 4.

[0137] Step 2.4: Build the backbone network, namely the HRNet network model, extract the multi-scale features of the extremely low-light image enhanced in step 2.3, classify and predict the positions of the joints, and predict the final image features of the human body posture estimation.

[0138] The backbone network is any CNN-based network, and the predicted joint point positions are determined by the heat map finally output by the backbone network.

[0139] like Figure 6 As shown in Figure 2, the loss function for training the extremely low-light human pose estimation network model uses the MSE loss function, which is expressed as follows:

[0140]

[0141] Among them, N represents the number of samples, W represents the number of types of human joints, Represents the predicted heat map of the wth human key point of the nth sample in a batch, Represents the heat map of the wth human key point label of the nth sample in a batch.

[0142] Step 3: Input the extremely low-light human image into the trained network model obtained in step 2 to obtain the human posture estimation result. Figure 7 As shown in the figure, the extremely low-light image to be detected is input into the image enhancement module to obtain richer and clearer image features; the enhanced image is input into the backbone network to obtain multi-scale features and finally obtain the predicted human posture estimation result.

[0143] This paper uses the ExLPose dataset to verify the effectiveness of the model. ExLPose is a low-light human pose estimation dataset used for the study of human pose estimation methods in extremely low light or low light. It contains a total of 2556 sets of low-light and normal-light image pairs under four different lighting conditions: extreme low light (LL-E), difficult low light (LL-H), normal low light (LL-N), and normal light (WL), and annotates 14214 human instances. Each human instance is manually annotated with a bounding box and 14 body joints. The dataset covers 251 real-world scenes such as streets and sports fields. Among them, 2065 pairs of images in 201 scenes are used for training, and 491 pairs of images in the remaining 50 scenes are used for testing.

[0144] To ensure a fair and comprehensive evaluation, the image size used for both training and testing was 256×192. During training, data augmentation such as random flipping and random scaling was used. The model was trained using the Adam optimizer with a linear learning rate and an equal interval adjustment learning strategy. The initial learning rate of the optimizer was 5×10 -4 , with a batch size of 64, and a total of 150 epochs of training. The model proposed in this application was implemented using the PyTorch framework, where the human pose estimation network is ResNet-50, pre-trained on the ImageNet dataset. All experiments were conducted on a single NVIDIA GeForce RTX3090 GPU.

[0145] This application follows the MSCOCO evaluation metric and uses object joint similarity (OKS) to calculate the average precision (AP) for evaluation results. The average mAP of multiple OKS values ​​(0.5 and 0.95) is used as the main metric.

[0146] In order to verify the effectiveness of the enhancement module, the features output by the enhancement module are grouped according to different lighting conditions for visualization. Figure 8 In the figure, Image is the original image, Scaled Image is the original image with pixel values ​​scaled to 0-255 in proportion for visualization, and the present invention is the feature map output by the enhancement module.

[0147] like Figure 8 As shown in the figure, the enhanced network can effectively capture the edge texture information of the relevant human body and extract the human body structure information, successfully retaining the potential features in low-light images that are conducive to detection, and has certain effects on low-light images under different lighting conditions.

[0148] Since the human pose estimation framework proposed in this paper is composed of a cascade of an enhancement module and a human pose estimation network, in order to verify the effectiveness of the enhancement network and its various structures for low-light images under different lighting conditions, the prediction results are grouped according to the lighting conditions and visualized.

[0149] Figure 9 Scaled Image is the original image with pixel values ​​scaled to 0-255 for intuitive comparison. GT is the true label of the data.

[0150] Baseline is the visualization of the results of training and prediction using only the human posture estimation network, HF is the visualization of training and prediction results by retaining only the high-frequency processing part in the enhancement module and cascading Resnet50, LF is the visualization of training and prediction results by retaining only the low-frequency processing part in the enhancement module and cascading Resnet50, and ours is the visualization of prediction results by using the network model of the present invention.

[0151] like Figure 9 As shown in Figure 2, as the lighting conditions of low-light images gradually deteriorate, the prediction results of the human pose estimation network alone become unreliable, and errors in the left-right relationship prediction of joints often occur, such as Figure 9 As shown in the prediction results at the 2nd row and 3rd column, the predictions for the left and right ankle joints are opposite. Especially in the case of extremely low light (LL-E), the prediction results are extremely unstable. When the cascaded pruned version of the human posture estimation network, that is, the enhancement module that only retains the high-frequency processing part HF or only retains the low-frequency processing part LF, the problem of incorrect prediction of the left and right relationship of the joints is alleviated, but the stability of the prediction results under extremely low light conditions needs to be improved. The extremely low-light human posture estimation framework proposed in the present invention not only effectively alleviates the problem of incorrect prediction of the left and right relationship of the joints, but also the prediction results are relatively stable and reliable under extremely low light conditions.

[0152] In summary, especially under more difficult extreme low-light conditions, the proposed method clearly shows the best results, and it can estimate the positions of human joints more accurately even under noisy LL-E conditions.

[0153] To verify the effectiveness of the various components of the enhancement model for processing images under varying lighting conditions, ablation experiments were conducted on the LL-A dataset. These experiments compared the performance of the low-frequency branch structure enhancement module, the high-frequency branch texture modulation module, the spatial alignment mechanism, and the cross-level guidance module. The improvements were measured for the BaseLine human pose estimation network.

[0154] The data in Table 1 shows that the processing of high-frequency texture information and low-frequency global structural information are both effective for low-light human pose estimation tasks. Since there is a spatial misalignment problem when fusing lower-resolution low-frequency components with higher-resolution high-frequency components, the spatial alignment reconstruction module effectively promotes the alignment and fusion of the two, further improving the model effect. The cross-level guidance module promotes information interaction between high- and low-frequency components. By introducing low-frequency guidance information, the low-frequency components can effectively guide the high-frequency components. As shown in the ablation experiment results in Table 1, the modules proposed in this invention are structurally reasonable and effective.

[0155] Table 1

[0156]

[0157] This application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium is capable of storing a computer program that, when executed by the data processing unit, executes the invention disclosure of a method for extremely low-light human pose estimation based on image frequency division feature enhancement provided by the present invention, as well as some or all of the steps in each embodiment. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0158] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes a number of instructions for enabling a device including a data processing unit (which can be a personal computer, server, single-chip microcomputer, MUU or network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0159] The network model proposed in this paper can use the frequency-aware image enhancement module to guide the network to generate richer and clearer image features, realize the coordinated optimization of image enhancement and human pose estimation, and thus improve the accuracy of extremely low-light human pose estimation.

[0160] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for extremely low-light human pose estimation based on image frequency division feature enhancement, characterized by: The extremely low-light human posture estimation method specifically comprises the following steps: Step 1: Collect extremely low-light human images to build training and test datasets; Step 2: Constructing an extremely low-light human pose estimation network model based on image frequency division feature enhancement. The extremely low-light human pose estimation network model includes a frequency-aware image enhancement module and a backbone network. The frequency-aware image enhancement module is used to enhance the training data set constructed in step 1. The enhanced data is then input into the backbone network for training to obtain a trained extremely low-light human pose estimation network model. The frequency-aware image enhancement module includes a Laplace decomposition layer, a texture modulation module, a structure enhancement module, and a spatial alignment reconstruction layer. The frequency-aware image enhancement module is used to decouple the high-frequency and low-frequency feature components of the extreme low-light image, and enhance the global structural information in the low-frequency feature components and the texture edge information in the high-frequency feature components respectively. The spatial alignment reconstruction ensures the effective fusion and structural consistency of the high-frequency and low-frequency feature components to obtain an enhanced extreme low-light image. The backbone network is used to extract multi-scale features of the enhanced extreme low-light image and predict the final human posture estimation. Step 3: Input the test data set into the trained network model obtained in step 2 to obtain the human posture estimation result, wherein the structure enhancement module includes channel frequency domain modulation based on energy compression and spatial frequency domain modulation based on parameter weighting. When the frequency-aware image enhancement module is used to enhance the training data in step 3, the structure enhancement module performs the following steps: Step 2.3.1: Input low-frequency feature component LF N Through 3D attention mechanism encoding, the low-frequency feature component LF of the input is re-weighted by optimizing the energy function N , for the low-frequency characteristic component LF N The feature map F∈R of the processed input C×H×W , calculate the mean according to the channel and variance Then calculate the energy value of each pixel by minimizing the energy function in the formula: Among them, λ is a hyperparameter, e ij Represents the target pixel t ij Energy value, i∈{0,1,…,H-1},j∈{0,1,…,W-1}, pixel weight Step 2.3.2, re-weight the points Adjust to weight in, represents the sum of the weight values ​​of each channel, and g(·) represents the activation function; Step 2.3.

3. Weight The spectrum graph f is further transformed into the frequency domain through discrete cosine transform (DCT) uv : f c =zigzag(f uv ) Among them, f uv Represents the value of the discrete cosine transformed spectrum f at the position (u,v), u∈{0,1,…,H-1}, v∈{0,1,…,W-1}, i, j refer to the position, f c is the frequency domain vector; Step 2.3.4, use zig-zag compression to convert the spectrum f uv Further compressed into a compact frequency domain vector f c ∈R C ×D and with a learnable filter Filter∈R C×H×W Perform element-wise multiplication to improve representation capability. The expression is as follows: f c1 =f c ·FC(R(Filter)) where, R C×D is converted from the filter parameter R by the Resize operation. The Resize operation includes the reshape operation R(·) and the linear layer FC(·). D < H×W, D is the dimension after zig-zag compression of H×W, and the channel weight f C×H×W is obtained by compression through the linear layer and weighted fused with the feature map F to obtain the enhanced feature F′ after channel frequency domain modulation: c2 ∈ R C , and weighted fusion with the feature map F to obtain the enhanced feature F′ after channel frequency domain modulation: F′=FC(f c1 )×F Where F′∈R C×H×W , FC(·) represents a linear layer, and × represents a multiplication operation along the channel; Step 2.3.5: Use the discrete cosine transform (DCT) operation to transform the input feature map F′∈R C×H×W Converted into frequency domain feature map f∈R C×H×W , using learnable filters to capture the importance relationship between frequency bands, indirectly interacting between different spatial locations of the image to obtain frequency domain information f * : f * =f·Filter Where Filter∈R C×H×W Represents the parameters of the learnable filter and is learned by the network; Step 2.3.6: frequency domain information f * Inverse conversion to spatial feature F″ ij : in F″ ij The value at position (i, j) of a channel feature map after inverse discrete cosine transform (IDCT) is the spatial domain feature.

2. The method for extremely low-light human pose estimation based on image frequency division feature enhancement according to claim 1, characterized in that: In step 2, building an extremely low-light human pose estimation network model based on image frequency division feature enhancement specifically includes the following steps: Step 2.1, extracting image features of different frequencies of the extreme low-light image and constructing a Laplace decomposition layer. The Laplace decomposition layer performs multi-scale frequency decomposition on the extracted extreme low-light image through a Laplace pyramid, decoupling the extreme low-light image features into high-frequency feature components and low-frequency feature components; Step 2.2: Construct a texture modulation module and a structure enhancement module. The texture modulation module adaptively learns the spatial weights of the fusion of high-frequency feature components at different scales, achieving complementarity and enhancement between high-frequency feature components to obtain enhanced high-frequency image features. The structure enhancement module consists of two parts: channel frequency domain modulation and spatial frequency domain modulation. They respectively model the dependencies between spatial frequency domain information and channel frequency domain information to obtain enhanced low-frequency image features. Step 2.3: Construct a spatial alignment reconstruction layer to fuse the enhanced high-frequency image features with the low-frequency image features based on the spatial alignment strategy to obtain an enhanced extreme low-light image. Step 2.4: Build the backbone network, namely the HRNet network model, extract the multi-scale features of the extremely low-light image enhanced in step 2.3, classify the joint points and predict their positions, and predict the final human posture estimation result.

3. The method for extremely low-light human pose estimation based on image frequency division feature enhancement according to claim 1, characterized in that: When using the frequency-aware image enhancement module to enhance training data, the Laplace decomposition layer performs the following steps: Step 2.1.

1. Use the Gaussian pyramid to construct a hierarchical structure for the extreme low-light image. The input extreme low-light image I with a resolution of h×w is decomposed into a low-frequency feature component and multiple high-frequency feature components: Among them, G n (I) represents the extremely low-light image at the nth level of the Gaussian pyramid, N represents the total number of pyramid levels, and B n (G n-1 (I)) represents the extremely low light image G of the previous layer n-1 (I) Perform 2D Gaussian filtering, where D(·) represents a 2x downsampling operation; Step 2.1.2: Repeated Gaussian filtering and downsampling filter out the high-frequency feature components and define the image at the highest level of the Gaussian pyramid as the low-frequency feature component LF. N : LF N =G N (I) Step 2.1.3: The high-frequency characteristic component HF decoupled from each layer of the Laplace pyramid n Obtained by subtracting two adjacent layers of Gaussian pyramid images: HF n =G n (I)-U(G n+1 (I)) Where 1≤n≤N-1, U(.) represents a 2x upsampling operation.

4. The method for extremely low-light human pose estimation based on image frequency division feature enhancement according to claim 1, characterized in that: The texture modulation module includes high-frequency feature scale unification, i.e., Resizing, and adaptive spatial weight learning and weighted fusion, which unifies high-frequency feature components of different scales to the same size. n The texture modulation module performs the following steps: Step 2.2.1, the texture modulation module first samples the high-frequency feature components of other branches to ensure that they are consistent with the size of the high-frequency feature components of the nth layer, and defines Represents the pixel value at (i, j) of the high-frequency component sampled from the n1th layer to the n2th layer; Step 2.2.2: Weight and fuse the different high-frequency feature components that are unified to the same size. The fusion strategy is expressed as follows: in, Represents the pixel value at position (i, j) in the high-frequency feature component after fusion of the nth layer, Refers to the spatial importance weights of the three different levels of high-frequency feature component feature maps of the nth layer. The corresponding weight map is obtained by using 1×1 convolution, and the sum of the spatial importance weights of the three different levels of high-frequency feature component feature maps is constrained to be 1: Through the fusion strategy, high-frequency feature components of all levels are adaptively fused at their respective scales.

5. The method for extremely low-light human pose estimation based on image frequency division feature enhancement according to claim 1, characterized in that: The spatial alignment reconstruction layer is implemented by a spatial alignment mechanism based on deformable convolution. When the frequency-aware image enhancement module is used to enhance the training data in step 3, the spatial alignment reconstruction layer performs the following steps: Step 2.4.1, adjust the low-frequency feature component according to the spatial position information provided by the high-frequency feature component, define the spatial position information adjustment amount as an offset, and represent it with a 2D feature map, where each offset value is regarded as a low-frequency feature component LF n Each point after upsampling is related to its HF n-1 The moving distance between the corresponding points in 2D space, the low-frequency feature component fusion reconstruction process expression is as follows: in, is the high-frequency feature component of the n-1th layer after spatial alignment reconstruction, f o (·),f a (·) denotes the offset Δ learned from the spatial difference n and a function to align features with offsets, f a (·), f o (·) implemented using deformable convolution, Step 2.4.2: For the fusion reconstruction of different high-frequency components, the following method is adopted: Δ n =f o ([HF n-2 ,HF′ n-1 ]), Where n = 1, 2, 3, 4.

6. The method for extremely low-light human pose estimation based on image frequency division feature enhancement according to claim 1, characterized in that: In step 2.4, the backbone network is any CNN-based network, and the predicted joint point positions are determined by the heat map finally output by the backbone network.

7. The method for extremely low-light human pose estimation based on image frequency division feature enhancement according to claim 1, characterized in that: The loss function for training the extremely low-light human pose estimation network model uses the MSE loss function, which is expressed as follows: Among them, N represents the number of samples, W represents the number of types of human joints, Represents the predicted heat map of the wth human key point of the nth sample in a batch, Represents the heat map of the wth human key point label of the nth sample in a batch.

Images