Extremely low light human body posture estimation method based on image frequency division feature enhancement

Abstract

The invention belongs to the technical field of human body posture estimation, and discloses an ultra-low light human body posture estimation method based on image frequency division feature enhancement, and the method comprises the following steps: collecting ultra-low light human body images to construct a training data set; constructing an ultra-low light human body posture estimation model based on image frequency division feature enhancement; in the training stage, training data is enhanced by using a frequency sensing image enhancement module, and then the training data is input to a network trunk for training to obtain a trained model; and a test stage: inputting a test data set into the trained network model to obtain a human body posture estimation result. The network model provided by the invention can guide the network to generate richer and clearer image features by using the frequency perception image enhancement module, and collaborative optimization of image enhancement and human body posture estimation is realized, so that the accuracy of extremely low light human body posture estimation is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human body posture estimation, and in particular relates to an extremely low-light human body posture estimation method based on image frequency division feature enhancement. Background Art

[0002] As one of the basic tasks of computer vision, human pose estimation aims to locate key points of the human body to obtain human pose information. It is widely used in action recognition, human-computer interaction, intelligent monitoring and other fields. In recent years, human pose estimation methods based on deep learning have made significant progress on multiple benchmark datasets. However, most existing studies focus on natural lighting environments, and there is little in-depth discussion on the problem of human pose estimation in extremely low light environments. In addition, the difference in image features in extremely low light environments puts forward more stringent requirements for human pose estimation. Taking extremely low light application scenarios such as nighttime autonomous driving and all-weather security monitoring as examples, their extremely low light images generally have the characteristics of reduced clarity and detail expression capabilities, which further leads to the existing human pose estimation models. The features extracted in extremely low light environments have problems such as structural confusion and unclear texture, making it difficult to accurately identify human joints.

[0003] Although previous studies have attempted to improve the performance of human pose estimation in extreme low light conditions by fine-tuning pre-trained models on extreme low light datasets or applying existing image enhancement techniques, these methods have significant limitations. First, simple fine-tuning strategies may affect the performance of the model under natural light conditions, which violates the principle of model generalization. Second, most existing extreme low light image enhancement techniques are dedicated to meeting the needs of human visual perception rather than optimizing for the needs of human pose estimation tasks, which results in only suboptimal benefits in improving the performance of extreme low light human pose estimation. In addition, some extreme low light image enhancement methods rely on strictly paired extreme low light and natural light images, but such paired data are often difficult to obtain in practical applications.

[0004] Therefore, a human body posture estimation method for extremely low-light images is urgently needed to improve the quality of extremely low-light images without relying on natural light images, thereby improving the detection accuracy of extremely low-light human body posture estimation. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an extremely low-light human posture estimation method based on image frequency division feature enhancement. The method constructs an extremely low-light human posture estimation model based on image frequency division feature enhancement, which significantly improves the robustness and accuracy of the human posture estimation model under different lighting conditions.

[0006] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0007] The present invention is a method for extremely low-light human posture estimation based on image frequency division feature enhancement, and the method specifically comprises the following steps:

[0008] Step 1: Collect extremely low-light human images to build training data sets and test data sets;

[0009] Step 2: construct an extremely low-light human posture estimation network model based on image frequency division feature enhancement, wherein the extremely low-light human posture estimation network model includes a frequency-aware image enhancement module and a backbone network, and the training data set constructed in step 1 is enhanced by using the frequency-domain-aware image enhancement module, and then the enhanced data is input into the backbone network for training to obtain a trained extremely low-light human posture estimation network model;

[0010] The frequency-aware image enhancement module includes a Laplace decomposition layer, a texture modulation module, a structure enhancement module and a spatial alignment reconstruction layer. The frequency-domain-aware image enhancement module is used to decouple the high-frequency feature components and the low-frequency feature components of the extremely low-light image, and enhance the global structure information in the low-frequency feature components and the texture edge information in the high-frequency feature components respectively. The effective fusion and structural consistency of the high-frequency feature components and the low-frequency feature components are ensured through spatial alignment reconstruction to obtain an enhanced extremely low-light image. The backbone network is used to extract multi-scale features of the enhanced extremely low-light image and predict the final human posture estimation result.

[0011] Step 3: Input the test data set into the trained network model obtained in step 2 to obtain the human body posture estimation result.

[0012] A further improvement of the present invention is that: Step 1 of collecting extremely low-light human images to construct a training data set is specifically: collecting extremely low-light human images from the low-light human image data set ExLPose to construct a training data set.

[0013] A further improvement of the present invention is that in step 2, constructing an extremely low-light human posture estimation network model based on image frequency division feature enhancement specifically includes the following steps:

[0014] Step 2.1, extracting image features of different frequencies of the extreme low light image, constructing a Laplace decomposition layer, wherein the Laplace decomposition layer performs multi-scale frequency decomposition on the extracted extreme low light image through a Laplace pyramid, and decouples the extreme low light image features into high-frequency feature components and low-frequency feature components;

[0015] Step 2.2, construct texture modulation module and structure enhancement module: the texture modulation module adaptively learns the spatial weights of the fusion of high-frequency feature components of different scales, realizes the complementarity and enhancement between high-frequency feature components, and obtains enhanced high-frequency image features. The structure enhancement module consists of two parts: channel frequency domain modulation and spatial frequency domain modulation, which respectively model the dependencies between spatial frequency domain information and channel frequency domain information, and obtains enhanced low-frequency image features to better capture the global structural information of the low-frequency components of the image;

[0016] Step 2.3, construct a spatial alignment reconstruction layer, perform a fusion operation on the enhanced high-frequency image features and the low-frequency image features based on the spatial alignment strategy, and obtain an enhanced extreme low-light image;

[0017] Step 2.4: Build the backbone network, namely the HRNet network model, extract the multi-scale features of the extremely low-light image enhanced in step 2.3, classify the joint points and predict their positions, and predict the final human posture estimation result.

[0018] A further improvement of the present invention is that when the frequency domain-aware image enhancement module is used to enhance the training data, the Laplace decomposition layer performs the following steps:

[0019] Step 2.1.1: Use the Gaussian pyramid to construct the extremely low-light image hierarchically. The input extremely low-light image with a resolution of h×w is decomposed into a low-frequency feature component and multiple high-frequency feature components:

[0020]

[0021] Among them, G n (I) represents the extremely low-light image at the nth level of the Gaussian pyramid, N represents the total number of pyramid levels, and B n (G n-1 (I)) represents the extremely low light image G of the previous layer n-1 (I) Perform 2D Gaussian filtering, where D(·) represents a 2x downsampling operation;

[0022] Step 2.1.2: Repeated Gaussian filtering and downsampling filter out the high-frequency feature components, and define the image at the highest level of the Gaussian pyramid as the low-frequency feature component LF. N :

[0023] LF N =G N (I)

[0024] Step 2.1.3: The high-frequency characteristic component HF decoupled from each layer of the Laplace pyramid n Obtained by subtracting two adjacent layers of Gaussian pyramid images:

[0025] HF n =Gn (I)-U(G n+1 (I))

[0026] Wherein, 1≤n≤N-1, U(.) represents a 2x upsampling operation.

[0027] A further improvement of the present invention is that the texture modulation module includes high-frequency feature scale unification, i.e., Resizing, and adaptive spatial weight learning and weighted fusion, i.e., unifying high-frequency feature components of different scales to the same size, and for the high-frequency feature component HF of the n-th layer branch n When the frequency domain-aware image enhancement module is used to enhance the training data, the texture modulation module performs the following steps:

[0028] Step 2.2.1: The texture modulation module first samples the high-frequency feature components of other branches to ensure that they are consistent with the size of the high-frequency feature components of the nth layer. Define Represents the pixel value of the high-frequency component at position (i, j) sampled from the n1th layer to the n2th layer;

[0029] Step 2.2.2: weight and fuse the different high-frequency feature components that are unified to the same size. Using the fact that the texture information of different high-frequency feature components has correlation in spatial distribution, the fusion strategy designed by the present invention is expressed as follows:

[0030]

[0031] in, Represents the pixel value at position (i, j) in the high-frequency feature component after fusion of the nth layer, Refers to the spatial importance weights of the feature maps of the three different levels of high-frequency feature components of the nth layer. The corresponding weight map is obtained by using 1×1 convolution, and the sum of the spatial importance weights of the feature maps of the three different levels of high-frequency feature components is constrained to be 1:

[0032]

[0033] Through the fusion strategy, high-frequency feature components of all levels are adaptively fused at their respective scales.

[0034] A further improvement of the present invention is that the structure enhancement module includes channel frequency domain modulation based on energy compression and spatial frequency domain modulation based on parameter weighting. When the frequency domain-aware image enhancement module is used to enhance the training data in step 3, the structure enhancement module performs the following steps:

[0035] Step 2.3.1: Input low-frequency feature component LF NEncoding through 3D attention mechanism, re-weighting the low-frequency feature component LF of the input by optimizing the energy function N , for the low-frequency characteristic component LF N The feature map F∈R of the processed input C ×H×W , calculate the mean according to the channel and variance Then the energy value of each pixel is calculated by minimizing the energy function in the formula:

[0036]

[0037] Among them, λ is a hyperparameter, e ij Represents the target pixel t ij Energy value, i∈{0, 1, ..., H-1}, j∈{0, 1, ..., W-1}, pixel weight

[0038]

[0039] Step 2.3.2: Re-weight the points Adjust to weight

[0040]

[0041] in, represents the sum of the weight values ​​of each channel, and g(·) represents the activation function;

[0042] Step 2.3.3. Weight The spectrum graph f is further transformed into the frequency domain through discrete cosine transform (DCT) uv :

[0043]

[0044] f c =zigzag(f uv )

[0045] Among them, f uv Represents the value of the discrete cosine transformed spectrum f at position (u, v), u∈{0,1,…,H-1}, v∈{0,1,…,W-1}, i, j refer to the position, f c is the frequency domain vector;

[0046] Step 2.3.4: Use zig-zag compression to convert the spectrum f uv Further compressed into a compact frequency domain vector f c ∈R C×D, and with the learnable filter Filter∈R C×H×W Perform element-wise multiplication to improve representation capability. The expression is as follows:

[0047] f c1 =f c FC(R(Filter))

[0048] Among them, R C×D The filter parameters are changed from R to C×H×W The Resize operation includes the reshape operation R(·) and the linear layer FC(·), D < H×W, D is the dimension of H×W after zig-zag compression, and the channel weight f is obtained by linear layer compression. c2 ∈R C , and weighted fused with the feature map F to obtain the feature F after channel frequency domain modulation enhancement ′ :

[0049] F ′ =FC(f c1 )×F

[0050] Among them, F ′ ∈R C×H×W , FC(·) represents a linear layer, and × represents a multiplication operation along the channel;

[0051] Step 2.3.5: Enhance the channel frequency domain modulation feature F ′ The discrete cosine transform (DCT) is used to transform it into frequency domain features and use a learnable filter to capture the weight relationship between frequency bands. The information of different frequency bands in the frequency domain features is weighted and fused. That is, the input feature map F is transformed into ′ ∈R C×H×W Transformed into frequency domain feature map f∈R C ×H×W , using learnable filters to capture the importance relationship between frequency bands, indirectly making more flexible interactions between different spatial locations of the image, and obtaining frequency domain information f * :

[0052] f * =f·Filter

[0053] Where Filter∈R C×H×W Represents the parameters of the learnable filter and is learned by the network;

[0054] Step 2.3.6: frequency domain information f * The inverse conversion is performed into spatial information for subsequent reconstruction. The feature map of a certain channel is processed, and the spatial frequency domain modulation expression is as follows:

[0055]

[0056] in F″ ij It indicates that the value of a channel feature map after inverse discrete cosine transform (IDCT) at position (i, j) is the enhanced feature after spatial frequency domain modulation.

[0057] A further improvement of the present invention is that the spatial alignment reconstruction layer is implemented by a spatial alignment mechanism based on deformable convolution, and when the frequency domain-aware image enhancement module is used to enhance the training data in step 3, the spatial alignment reconstruction layer performs the following steps:

[0058] Step 2.4.1, adjust the low-frequency feature component according to the spatial position information provided by the high-frequency feature component, define the spatial position information adjustment amount as an offset, and represent it with a 2D feature map, where each offset value is regarded as a low-frequency feature component LF n Each point after upsampling is related to its HF n-1 The moving distance between the corresponding points in 2D space, the low-frequency feature component fusion reconstruction process expression is as follows:

[0059]

[0060] in, is the high-frequency feature component of the n-1th layer after spatial alignment reconstruction, f o (·),f a (·) respectively represent the offset Δ learned from the spatial difference n and a function to align features with offsets, f a (·), f o (·) implemented using deformable convolution,

[0061] Step 2.4.2: The fusion reconstruction of different high-frequency components is completed in the following way:

[0062]

[0063] Where n=1,2,3,4.

[0064] A further improvement of the present invention is that in step 2.4, the backbone network is any CNN-based network, and the predicted joint point positions are determined by the heat map finally output by the backbone network.

[0065] A further improvement of the present invention is that the loss function for training the extremely low-light human posture estimation network model uses the MSE loss function, which is expressed as follows:

[0066]

[0067] Where N represents the number of samples, W represents the number of human joint types, Represents the predicted heat map of the wth human key point of the nth sample in a batch, Represents the heat map of the wth human key point label of the nth sample in a batch.

[0068] The beneficial effects of the present invention are:

[0069] The method of the present invention proposes an image enhancement paradigm that shifts the problem framework from human visual perception to machine perception, provides a directly relevant information enhancement strategy for the task of human pose estimation in extremely low light environments, breaks away from the limitations of strictly paired natural light images, and achieves collaborative optimization of image enhancement and human pose estimation.

[0070] Based on the characteristic that different frequency components of an image have different physical meanings, the method of the present invention extracts and enhances task-related robust features from a frequency perspective, designs an image enhancement module based on frequency perception, and seamlessly integrates it into the front end of a human pose estimation network, thereby constructing an extremely low-light human pose estimation model based on image frequency division feature enhancement. This avoids the tedious alignment processing of traditional domain adaptive methods, enables the model to maintain a high computational efficiency under various lighting conditions, and significantly improves the robustness and accuracy of the human pose estimation model under different lighting conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the workflow of the extremely low-light human posture estimation method of the present invention.

[0072] Figure 2 It is a network structure diagram of the image enhancement module of the present invention.

[0073] Figure 3 It is a network structure diagram of the texture modulation module in the image enhancement module of the present invention.

[0074] Figure 4 It is a network structure diagram of the structure enhancement module in the image enhancement module of the present invention.

[0075] Figure 5 It is a network structure diagram of the spatial alignment reconstruction layer in the image enhancement module of the present invention.

[0076] Figure 6 It is a schematic diagram of the workflow of the training phase of the present invention.

[0077] Figure 7 It is a schematic diagram of the workflow of the testing phase of the present invention.

[0078] Figure 8 A comparison chart for visualization based on different lighting conditions.

[0079] Fig. 9 This is a visualization comparison of low-light image prediction results under different lighting conditions. DETAILED DESCRIPTION

[0080] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0081] like Figure 1 As shown, the present invention provides an extremely low-light human posture estimation method based on image frequency division feature enhancement, which adds an image enhancement module on the basis of the original backbone network, thereby making it easier to perform human posture estimation in aurora scenes and improving the accuracy of human posture estimation, and specifically includes the following steps:

[0082] Step 1. Collect extremely low-light human images to construct a training dataset. Specifically, collect extremely low-light human images from the low-light human image dataset ExLPose to construct a training dataset and a test dataset, where the ratio of the training dataset to the test dataset is 4:1.

[0083] Step 2: construct an extremely low-light human posture estimation network model based on image frequency division feature enhancement, wherein the extremely low-light human posture estimation network model includes a frequency-aware image enhancement module and a backbone network, and the training data set constructed in step 1 is enhanced by using the frequency-domain-aware image enhancement module, and then the enhanced data is input into the backbone network for training to obtain a trained extremely low-light human posture estimation network model;

[0084] Among them, the frequency-aware image enhancement module includes a Laplace decomposition layer, a texture modulation module, a structure enhancement module and a spatial alignment reconstruction layer. The frequency-domain-aware image enhancement module is used to decouple the high-frequency feature components and the low-frequency feature components of the extremely low-light image, and enhance the global structural information in the low-frequency feature components and the texture edge information of the high-frequency feature components respectively. The effective fusion and structural consistency of the high-frequency feature components and the low-frequency feature components are ensured through spatial alignment reconstruction to obtain an enhanced extremely low-light image. The backbone network is used to extract the multi-scale features of the enhanced extremely low-light image and predict the final human posture estimation result.

[0085] like Figure 2 As shown in FIG. 1 , constructing an extremely low-light human posture estimation network model based on image frequency division feature enhancement specifically includes the following steps:

[0086] Step 2.1, extract image features of different frequencies of the extreme low light image, and construct a Laplace decomposition layer so as to subsequently enhance and fuse these features. The Laplace decomposition layer performs multi-scale frequency decomposition on the extracted extreme low light image through a Laplace pyramid, and decouples the extreme low light image features into high-frequency feature components and low-frequency feature components.

[0087] When the frequency-domain-aware image enhancement module is used to enhance the training data, the Laplace decomposition layer performs the following steps:

[0088] Step 2.1.1: Use the Gaussian pyramid to construct the extremely low-light image hierarchically. The input extremely low-light image with a resolution of h×w is decomposed into a low-frequency feature component and multiple high-frequency feature components:

[0089]

[0090] Among them, G n (I) represents the extremely low-light image at the nth level of the Gaussian pyramid, N represents the total number of pyramid levels, and B n (G n-1 (I)) represents the extremely low light image G of the previous layer n-1 (I) Perform 2D Gaussian filtering, where D(·) represents a 2x downsampling operation;

[0091] Step 2.1.2: Since repeated Gaussian filtering and downsampling can filter out high-frequency feature components, the image at the highest level of the Gaussian pyramid can be defined as the low-frequency feature component LF. N :

[0092] LF N =G N (I)

[0093] Step 2.1.3: The high-frequency characteristic component HF decoupled from each layer of the Laplace pyramid n Obtained by subtracting two adjacent layers of Gaussian pyramid images:

[0094] HF n =G n (I)-U(G n+1 (I))

[0095] Wherein, 1≤n≤N-1, U(.) represents a 2x upsampling operation.

[0096] Step 2.2, construct a texture modulation module and a structure enhancement module: used to enhance the image features of different frequencies of the extremely low-light image respectively, so as to reconstruct image features that are more conducive to the extremely low-light human posture estimation network model to perform human posture estimation; the texture modulation module adaptively learns the spatial weights of the fusion of high-frequency feature components of different scales, realizes the complementarity and enhancement between high-frequency feature components, and obtains enhanced high-frequency image features. The structure enhancement module consists of two parts: channel frequency domain modulation and spatial frequency domain modulation, which respectively model the dependencies between spatial frequency domain information and channel frequency domain information to obtain enhanced low-frequency image features, so as to better capture the global structural information of the low-frequency components of the image.

[0097] like Figure 3 As shown, the texture modulation module includes high-frequency feature scale unification, i.e., Resizing, and adaptive spatial weight learning and weighted fusion, that is, unifying high-frequency feature components of different scales to the same size. When the frequency-domain-aware image enhancement module is used to enhance the training data, the texture modulation module performs the following steps:

[0098] Step 2.2.1: High frequency characteristic component HF of the nth layer branch n The texture modulation module first samples the high-frequency feature components of other branches to ensure that they are consistent with the size of the high-frequency feature components of the nth layer, and defines Represents the pixel value of the high-frequency component at position (i, j) sampled from the n1th layer to the n2th layer;

[0099] Step 2.2.2: weight and fuse the different high-frequency feature components that are unified to the same size. Using the fact that the texture information of different high-frequency feature components has correlation in spatial distribution, the fusion strategy designed by the present invention is expressed as follows:

[0100]

[0101] in, Represents the pixel value at position (i, j) in the high-frequency feature component after fusion of the nth layer, Refers to the spatial importance weights of the feature maps of the three different levels of high-frequency feature components of the nth layer. The corresponding weight map is obtained by using 1×1 convolution, and the sum of the spatial importance weights of the feature maps of the three different levels of high-frequency feature components is constrained to be 1:

[0102]

[0103] Through the fusion strategy, high-frequency feature components of all levels are adaptively fused at their respective scales.

[0104] like Figure 4As shown, the structure enhancement module includes channel frequency domain modulation based on energy compression and spatial frequency domain modulation based on parameter weighting. When the frequency domain-aware image enhancement module is used to enhance the training data in step 3, the structure enhancement module performs the following steps:

[0105] Step 2.3.1: Input low-frequency feature component LF N Encoding through 3D attention mechanism, re-weighting the low-frequency feature component LF of the input by optimizing the energy function N , for the low-frequency characteristic component LF N The feature map F∈R of the processed input C ×H×W , calculate the mean according to the channel and variance Then the energy value of each pixel is calculated by minimizing the energy function in the formula:

[0106]

[0107] Among them, λ is a hyperparameter, which is empirically set to 0.0001, e ij Represents the target pixel t ij Energy value, i∈{0, 1, ..., H-1}, j∈{0, 1, ..., W-1}, pixel weight

[0108]

[0109] Among them, F∈R C×H×W 3*32*24, H=32, W=24

[0110] Step 2.3.2: In order to prevent excessive weights from causing instability in training, use the idea of ​​normalization to re-adjust the weights of the pixels. Adjust to weight

[0111]

[0112] in, represents the sum of the weight values ​​of each channel, g(·) represents the activation function; the present invention adopts the LeakyReLU activation function.

[0113] Step 2.3.3. Weight The spectrum graph f is further transformed into the frequency domain through discrete cosine transform (DCT) uv :

[0114]

[0115] f c =zigzag(fuv )

[0116] Among them, f uv Represents the value of the discrete cosine transformed spectrum f at position (u, v), u∈{0,1,…,H-1}, v∈{0,1,…,W-1}, i, j refer to the position, f c is the frequency domain vector;

[0117] Step 2.3.4: Then, referring to the JPEG standard, the spectrum image f is compressed using the zig-zag compression method. uv Further compressed into a compact frequency domain vector f c ∈R C×D , and with the learnable filter Filter∈R C×H×W Perform element-wise multiplication to improve representation capability. The expression is as follows:

[0118] f c1 =f c FC(R(Filter))

[0119] Figure 3 In the example, the Resize operation includes the reshape operation R(·) and the linear layer FC(·), which transforms the filter parameters from R C ×H×W Convert R C×D , where D<H×W, D is the dimension of H×W after zig-zag compression, and the present invention takes D=16. The channel weight f is obtained by linear layer compression c2 ∈R C , and weighted fused with the feature map F to obtain the feature F after channel frequency domain modulation enhancement ′ :

[0120] F ′ =FC(f c1 )×F

[0121] Among them, F ′ ∈R C×H×W , FC(·) represents a linear layer, and × represents a multiplication operation along the channel;

[0122] Step 2.3.5: Enhance the channel frequency domain modulation feature F ′ The discrete cosine transform (DCT) is used to transform it into frequency domain features and use a learnable filter to capture the weight relationship between frequency bands. The information of different frequency bands in the frequency domain features is weighted and fused. That is, the input feature map F is transformed into ′ ∈R C×H×W Transformed into frequency domain feature map f∈R C ×H×W, using learnable filters to capture the importance relationship between frequency bands, indirectly making more flexible interactions between different spatial locations of the image, and obtaining frequency domain information f * :

[0123] f * =f·Filter

[0124] Where Filter∈R C×H×W Represents the parameters of the learnable filter and is learned by the network;

[0125] Step 2.3.6: frequency domain information f * The inverse conversion is performed into spatial information for subsequent reconstruction. The feature map of a certain channel is processed, and the spatial frequency domain modulation expression is as follows:

[0126]

[0127] in F″ ij It indicates that the value of a channel feature map after inverse discrete cosine transform (IDCT) at position (i, j) is the enhanced feature after spatial frequency domain modulation.

[0128] Step 2.3: construct a spatial alignment reconstruction layer, and fuse the enhanced high-frequency image features and low-frequency image features based on the spatial alignment strategy to obtain an enhanced extreme low-light image.

[0129] like Figure 5 As shown, the spatial alignment reconstruction layer is implemented by a spatial alignment mechanism based on deformable convolution. When the frequency-domain-aware image enhancement module is used to enhance the training data in step 3, the spatial alignment reconstruction layer performs the following steps:

[0130] Step 2.4.1, adjust the low-frequency feature component according to the spatial position information provided by the high-frequency feature component, define the spatial position information adjustment amount as an offset, and represent it with a 2D feature map, where each offset value is regarded as a low-frequency feature component LF n Each point after upsampling is related to its HF n-1 The moving distance between the corresponding points in 2D space, the low-frequency feature component fusion reconstruction process expression is as follows:

[0131]

[0132]

[0133] in, is the high-frequency feature component of the n-1th layer after spatial alignment reconstruction, f o (·),f a (·) respectively represent the offset Δ learned from the spatial differencen and a function to align features with offsets, f a (·), f o (·) implemented using deformable convolution,

[0134] Step 2.4.2: The fusion reconstruction of different high-frequency components is completed in the following way:

[0135]

[0136] Where n=1,2,3,4.

[0137] Step 2.4: Build the backbone network, namely the HRNet network model, extract the multi-scale features of the extremely low-light image enhanced in step 2.3, classify and predict the positions of the joints, and predict the final image features of the human body posture estimation.

[0138] The backbone network is any CNN-based network, and the predicted joint point positions are determined by the heat map finally output by the backbone network.

[0139] like Figure 6 As shown, the loss function for training the extremely low-light human pose estimation network model uses the MSE loss function, which is expressed as follows:

[0140]

[0141] Where N represents the number of samples, W represents the number of human joint types, Represents the predicted heat map of the wth human key point of the nth sample in a batch, Represents the heat map of the wth human key point label of the nth sample in a batch.

[0142] Step 3: Input the extremely low-light human image into the trained network model obtained in step 2 to obtain the human posture estimation result. Figure 7 As shown, the extremely low-light image to be detected is input into the image enhancement module to obtain richer and clearer image features; the enhanced image is input into the backbone network to obtain multi-scale features and finally obtain the predicted human posture estimation result.

[0143] The present invention uses the ExLPose dataset to verify the effectiveness of the model. ExLPose is a low-light human pose estimation dataset used for the study of extremely low-light or low-light human pose estimation methods. It contains a total of 2556 sets of low-light and normal-light image pairs under four different lighting conditions: extremely low light (LL-E), difficult low light (LL-H), normal low light (LL-N) and normal light (WL), and annotates 14214 human instances. Each human instance is manually annotated with a bounding box and 14 body joints. The dataset involves 251 real-life scenes such as streets and stadiums. 2065 pairs of images in 201 scenes are used for training, and 491 pairs of images in the remaining 50 scenes are used for testing.

[0144] To ensure a fair and comprehensive evaluation, the image size used for training and testing is 256×192. During the training process, data augmentation such as random flipping and random scaling is used. The model is trained using the Adam optimizer with a linear learning rate and an equal interval adjustment learning strategy. The initial learning rate of the optimizer is 5×10 -4 , the batch size is 64, and a total of 150 epochs are trained. The model proposed in this application is implemented using the Pytorch framework, where the human pose estimation network is resnet-50 and pre-trained on the ImageNet dataset. All experiments are performed on a single NVIDIA GeForce RTX3090 GPU.

[0145] This application follows the evaluation indicators of MSCOCO and uses the object joint point similarity (OKS) to calculate the average precision (AP) for evaluation results. The average mAP of multiple OKS values ​​(0.5:0.95) is used as the main indicator.

[0146] In order to verify the effectiveness of the enhancement module, the features output by the enhancement module are grouped according to different lighting conditions for visualization. Figure 8 In the figure, Image is the original image, Scaled Image is the original image with pixel values ​​scaled to 0-255 in proportion for visualization, and the present invention is the feature map output by the enhancement module.

[0147] like Figure 8 As shown in the figure, the enhanced network can effectively capture the edge texture information of the relevant human body and extract the human body structure information, successfully retaining the potential features in low-light images that are conducive to detection, and has a certain effect on low-light images under different lighting conditions.

[0148] Since the human pose estimation framework proposed in this paper is composed of a cascade of an enhancement module and a human pose estimation network, in order to verify the effectiveness of the enhancement network and its various structures for low-light images under different lighting conditions, the prediction results are grouped according to the lighting conditions for visualization.

[0149] Fig. 9 Scaled Image is the original image with pixel values ​​scaled to 0-255 for intuitive comparison. GT is the true label of the data.

[0150] Baseline is the visualization of the results of training and prediction using only the human posture estimation network, HF is to retain only the high-frequency processing part in the enhancement module and cascade Resnet50 for training and visualization of the prediction results, LF is to retain only the low-frequency processing part in the enhancement module and cascade Resnet50 for training and visualization of the prediction results, and ours is to use the network model of the present invention for visualization of the prediction results.

[0151] like Fig. 9 As shown in Figure 1, as the lighting conditions of low-light images gradually deteriorate, the prediction results of the human pose estimation network alone become unreliable, and the left-right relationship of joints is often mispredicted, such as Fig. 9 As shown in the prediction result located in the 2nd row and 3rd column, the predictions for the left and right ankle joints are opposite. Especially in the case of extremely low light (LL-E), the prediction results are extremely unstable. When the cascaded deleted version of the human posture estimation network, that is, only retaining the high-frequency processing part HF or only retaining the enhancement module of the low-frequency processing part LF, the problem of incorrect prediction of the left and right relationship of the joints is alleviated, but the stability of the prediction results under extremely low light conditions needs to be improved. The extremely low light human posture estimation framework proposed in the present invention not only effectively alleviates the problem of incorrect prediction of the left and right relationship of the joints, but also the prediction results are relatively stable and reliable under extremely low light conditions.

[0152] In summary, especially under more difficult extreme low-light conditions, the method of the present invention clearly shows the best results, and it can estimate the positions of human joints more accurately even under noisy LL-E conditions.

[0153] In order to verify the effectiveness of each part of the enhancement model in processing images under different lighting conditions, ablation experiments were conducted on the LL-A dataset. The ablation experiments compared the effects of the structure enhancement module of the low-frequency branch, the texture modulation module of the high-frequency branch, the spatial alignment mechanism, and the cross-level guidance module. The improvement effects were all for the BaseLine network of human pose estimation.

[0154] From the data in Table 1, it can be seen that the processing of high-frequency texture information and the processing of low-frequency global structure information are both effective for low-light human posture estimation tasks. Since there is a spatial misalignment problem when the low-resolution low-frequency components and the high-resolution high-frequency components are fused, the spatial alignment reconstruction module effectively promotes the alignment and fusion of the two, further improving the model effect. The cross-level guidance module promotes the information interaction of high- and low-frequency components, and by introducing low-frequency guidance information, the low-frequency components can effectively guide the high-frequency components. As shown in the ablation experiment results in Table 1, the modules proposed in the present invention are reasonable and effective.

[0155] Table 1

[0156]

[0157] The present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of the method for extremely low-light human posture estimation based on image frequency division feature enhancement provided by the present invention and some or all steps in each embodiment can be run. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0158] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention are essentially or partly contributed to the prior art can be embodied in the form of a computer program, i.e., a software product, which can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0159] The network model proposed in the present invention can utilize the frequency-aware image enhancement module to guide the network to generate richer and clearer image features, realize the coordinated optimization of image enhancement and human posture estimation, and thus improve the accuracy of extremely low-light human posture estimation.

[0160] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for extremely low-light human posture estimation based on image frequency division feature enhancement, characterized in that: The extremely low-light human body posture estimation method specifically comprises the following steps: Step 1: Collect extremely low-light human images to build training data sets and test data sets; Step 2: construct an extremely low-light human posture estimation network model based on image frequency division feature enhancement, wherein the extremely low-light human posture estimation network model includes a frequency-aware image enhancement module and a backbone network, and the training data set constructed in step 1 is enhanced by using the frequency-domain-aware image enhancement module, and then the enhanced data is input into the backbone network for training to obtain a trained extremely low-light human posture estimation network model; The frequency-aware image enhancement module includes a Laplace decomposition layer, a texture modulation module, a structure enhancement module and a spatial alignment reconstruction layer. The frequency-domain-aware image enhancement module is used to decouple the high-frequency feature components and the low-frequency feature components of the extremely low-light image, and enhance the global structure information in the low-frequency feature components and the texture edge information in the high-frequency feature components respectively. The effective fusion and structural consistency of the high-frequency feature components and the low-frequency feature components are ensured through spatial alignment reconstruction to obtain an enhanced extremely low-light image. The backbone network is used to extract multi-scale features of the enhanced extremely low-light image and predict the final human posture estimation. Step 3: Input the test data set into the trained network model obtained in step 2 to obtain the human body posture estimation result.

2. The method for extremely low-light human posture estimation based on image frequency division feature enhancement according to claim 1, characterized in that: In step 2, building an extremely low-light human posture estimation network model based on image frequency division feature enhancement specifically includes the following steps: Step 2.1, extracting image features of different frequencies of the extreme low light image, constructing a Laplace decomposition layer, wherein the Laplace decomposition layer performs multi-scale frequency decomposition on the extracted extreme low light image through a Laplace pyramid, and decouples the extreme low light image features into high-frequency feature components and low-frequency feature components; Step 2.2, construct texture modulation module and structure enhancement module: the texture modulation module adaptively learns the spatial weights of the fusion of high-frequency feature components of different scales, realizes the complementarity and enhancement between high-frequency feature components, and obtains the enhanced high-frequency image features. The structure enhancement module consists of two parts: channel frequency domain modulation and spatial frequency domain modulation, which respectively model the dependencies between spatial frequency domain information and channel frequency domain information to obtain the enhanced low-frequency image features. Step 2.3, construct a spatial alignment reconstruction layer, perform a fusion operation on the enhanced high-frequency image features and the low-frequency image features based on the spatial alignment strategy, and obtain an enhanced extreme low-light image; Step 2.4: Build the backbone network, namely the HRNet network model, extract the multi-scale features of the extremely low-light image enhanced in step 2.3, classify the joint points and predict their positions, and predict the final human posture estimation result.

3. The method for extremely low-light human posture estimation based on image frequency division feature enhancement according to claim 1, characterized in that: When the frequency-domain-aware image enhancement module is used to enhance the training data, the Laplace decomposition layer performs the following steps: Step 2.1.1: Use the Gaussian pyramid to construct the extremely low-light image hierarchically. The input extremely low-light image I with a resolution of h×w is decomposed into a low-frequency feature component and multiple high-frequency feature components: Among them, G n (I) represents the extremely low-light image at the nth level of the Gaussian pyramid, N represents the total number of pyramid levels, and B n (G n-1 (I)) represents the extremely low light image G of the previous layer n-1 (I) Perform 2D Gaussian filtering, where D(·) represents a 2x downsampling operation; Step 2.1.2: Repeated Gaussian filtering and downsampling filter out the high-frequency feature components, and define the image at the highest level of the Gaussian pyramid as the low-frequency feature component LF. N : LF N =G N (I) Step 2.1.3: The high-frequency characteristic component HF decoupled from each layer of the Laplace pyramid n Obtained by subtracting two adjacent layers of Gaussian pyramid images: HF n =G n (I)-U(G n+1 (I)) Where 1≤n≤N-1, U(·) represents a 2x upsampling operation.

4. The method for extremely low-light human posture estimation based on image frequency division feature enhancement according to claim 1, characterized in that: The texture modulation module includes high-frequency feature scale unification, i.e., resizing, and adaptive spatial weight learning and weighted fusion, that is, unifying high-frequency feature components of different scales to the same size, and for the high-frequency feature component HF of the n-th layer branch n The texture modulation module performs the following steps: Step 2.2.1: The texture modulation module first samples the high-frequency feature components of other branches to ensure that they are consistent with the size of the high-frequency feature components of the nth layer. Define Represents the pixel value of the high-frequency component at position (i, j) sampled from the n1th layer to the n2th layer; Step 2.2.2: Weight and fuse the different high-frequency feature components that are unified to the same size. The fusion strategy is expressed as follows: in, Represents the pixel value at position (i, j) in the high-frequency feature component after fusion of the nth layer, Refers to the spatial importance weights of the feature maps of the three different levels of high-frequency feature components of the nth layer. The corresponding weight map is obtained by using 1×1 convolution, and the sum of the spatial importance weights of the feature maps of the three different levels of high-frequency feature components is constrained to be 1: Through the fusion strategy, high-frequency feature components of all levels are adaptively fused at their respective scales.

5. The method for extremely low-light human posture estimation based on image frequency division feature enhancement according to claim 1, characterized in that: The structure enhancement module includes channel frequency domain modulation based on energy compression and spatial frequency domain modulation based on parameter weighting. When the frequency domain-aware image enhancement module is used to enhance the training data in step 3, the structure enhancement module performs the following steps: Step 2.3.1: Input low-frequency feature component LF N Through 3D attention mechanism encoding, the low-frequency feature component LF of the input is re-weighted by optimizing the energy function N , for the low-frequency characteristic component LF N The feature map F∈R of the processed input C×H×W , calculate the mean according to the channel and variance Then the energy value of each pixel is calculated by minimizing the energy function in the formula: Among them, λ is a hyperparameter, e ij Represents the target pixel t ij Energy value, i∈{0, 1, ..., H-1}, j∈{0, 1, ..., W-1}, pixel weight Step 2.3.2: Re-weight the points Adjust to weight in, represents the sum of the weight values ​​of each channel, and g(·) represents the activation function; Step 2.3.

3. Weight The spectrum graph f is further transformed into the frequency domain through discrete cosine transform (DCT) uv : f c =zigzag(f uv ) Among them, f uv Represents the value of the discrete cosine transformed spectrum f at position (u, v), u∈{0,1,…,H-1}, v∈{0,1,…,W-1}, i, j refer to the position, f c is the frequency domain vector; Step 2.3.4: Use zig-zag compression to convert the spectrum f uv Further compressed into a compact frequency domain vector f c ∈R C ×D , and with the learnable filter Filter∈R C×H×W Perform element-wise multiplication to improve representation capability. The expression is as follows: f c1 =f c ·FC(R(Filter)) Among them, R C×D is converted from the filter parameters by the Resize operation. The Resize operation includes the reshape operation R(·) and the linear layer FC(·). D<H×W, where D is the dimension after zig-zag compression of H×W, and the channel weight f C×H×W is obtained by compression through the linear layer. c2 ∈R C , and is weighted and fused with the feature map F to obtain the enhanced feature F′ after channel frequency domain modulation: F′=FC(f c1 )×F Among them, F′∈R C×H×W , FC(·) represents a linear layer, and × represents a multiplication operation along the channel; Step 2.3.5: Use the discrete cosine transform (DCT) operation to transform the input feature map F′∈R C×H×W Transformed into frequency domain feature map f∈R C×H×W , using learnable filters to capture the importance relationship between frequency bands, indirectly interacting between different spatial locations of the image to obtain frequency domain information f * : f * =f·Filter Among them, Filter∈R C×H×W Represents the parameters of the learnable filter and is learned by the network; Step 2.3.6: frequency domain information f * Inverse conversion to spatial domain feature F″ ij : in F″ ij The value representing the position (i, j) of a channel feature map after the inverse discrete cosine transform (IDCT) is the spatial domain feature.

6. The method for extremely low-light human posture estimation based on image frequency division feature enhancement according to claim 5, characterized in that: The spatial alignment reconstruction layer is implemented by a spatial alignment mechanism based on deformable convolution. When the frequency-domain-aware image enhancement module is used to enhance the training data in step 3, the spatial alignment reconstruction layer performs the following steps: Step 2.4.1, adjust the low-frequency feature component according to the spatial position information provided by the high-frequency feature component, define the spatial position information adjustment amount as an offset, and represent it with a 2D feature map, where each offset value is regarded as a low-frequency feature component LF n Each point after upsampling is related to its HF n-1 The moving distance between the corresponding points in 2D space, the low-frequency feature component fusion reconstruction process expression is as follows: in, is the high-frequency feature component of the n-1th layer after spatial alignment reconstruction, f o (·),f a (·) respectively represent the offset Δ learned from the spatial difference n and a function to align features with offsets, f A (·), f o (·) implemented using deformable convolution, Step 2.4.2: The fusion reconstruction of different high-frequency components is completed in the following way: Δ n =f o ([HF n-2， HF′ n-1 ]), Where n=1,2,3,4.

7. The method for extremely low-light human posture estimation based on image frequency division feature enhancement according to claim 1, characterized in that: In step 2.4, the backbone network is any CNN-based network, and the predicted joint point positions are determined by the heat map finally output by the backbone network.

8. The method for extremely low-light human posture estimation based on image frequency division feature enhancement according to claim 1, characterized in that: The loss function for training the extremely low-light human pose estimation network model uses the MSE loss function, which is expressed as follows: Among them, N represents the number of samples, W represents the number of types of human joints, Represents the predicted heat map of the wth human key point of the nth sample in a batch, Represents the heat map of the wth human key point label of the nth sample in a batch.

Images