High-precision line of sight estimation method in low-light environment

Abstract

The present invention discloses a high-precision line of sight estimation method in a low-light environment, comprising the following steps: S1. Data set preprocessing, simulating a low-light environment S2. Low-light image addition to obtain an enhanced image; S3. Enhanced image calibration to obtain a calibrated image; S4. Image feature extraction, output feature vector; Use the improved residual network model ResNet18 to extract features from the calibrated image; S5. Map the feature vector into a three-dimensional output vector through a fully connected layer; S6. Apply a hyperbolic tangent transformation to the first two elements of the three-dimensional output vector to obtain an accurate predicted line of sight direction; S7. Transform the third element of the three-dimensional output vector through a sigmoid function to obtain the uncertainty of the line of sight prediction; S8. Use the MSELoss loss function to measure the error between the predicted result and the true value. The method of the present invention can effectively solve the problem of significant decrease in the accuracy of line of sight estimation in a low-light environment, thereby improving the practicality and accuracy of the system.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and computer vision, and in particular to a high-precision line of sight estimation method in a low-light environment. Background Art

[0002] Gaze estimation technology aims to determine the gaze direction and focus of a person in an image or video. As a core element of human social interaction, gaze behavior can reveal a lot of potential information. Traditional gaze estimation methods mainly rely on model-driven technology and require the use of expensive high-resolution and infrared cameras and other complex equipment. However, with the rapid development of appearance-based gaze estimation technology, especially the method of directly estimating the gaze position from face or eye images by using convolutional neural networks (CNN), major technical breakthroughs have been made. These methods mainly adopt end-to-end learning strategies, and can achieve accurate gaze estimation by only inputting a face image by mapping the image to a function of the gaze direction. This method not only simplifies the data processing process, but also significantly improves the accuracy and practicality of gaze estimation by comprehensively analyzing the global features of the whole face. It has become the mainstream technology in the field of gaze estimation.

[0003] Although these innovative methods have achieved remarkable results in cross-domain adaptation, fine-grained processing, multi-person recognition, real-time applications and robustness, it is still a challenge to achieve high-precision line of sight estimation in low-light scenes. Therefore, the present invention is committed to improving the performance and accuracy of line of sight estimation in low-light environments to meet the increasingly stringent requirements of practical applications.

[0004] In order to solve the high-precision problem of line of sight estimation, a method using dual-view interactive convolution blocks and dual-view TransFormer to exchange and fuse features has been proposed in the prior art. The line of sight estimation network of this method accepts face images from two different perspectives as input, and uses TransFormer to exchange and fuse features to complete the line of sight estimation task. However, this method shows its limitations when facing various complex low-light scenes.

[0005] Therefore, a new high-precision line of sight estimation method in low-light environments is urgently needed. Summary of the invention

[0006] The implementation purpose of the present invention is to provide a high-precision line of sight estimation method in a low-light environment, and to design and introduce an innovative ALGCF (adaptive local-global fusion) module into the low-light image enhancement network to specifically address the importance of eye features and their dynamic changes in the line of sight estimation task. The ALGCF module provides an effective multi-scale fusion strategy through an adaptive fusion gating mechanism, combined with a local feature extractor and a global context information extractor, which greatly improves the accuracy. In addition, based on low-light image enhancement technology, the present invention designs a line of sight estimation model that integrates a feature purification module and an attention mechanism. This design ensures that eye features can be effectively extracted even under low-light conditions, thereby performing accurate line of sight estimation. The method of the present invention can effectively solve the problem of a significant decrease in the accuracy of line of sight estimation in a low-light environment, thereby improving the practicality and accuracy of the system. In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0007] In order to solve at least one of the above technical problems, according to one aspect of the present invention, a high-precision line of sight estimation method in a low-light environment specifically includes the following steps:

[0008] S1. Dataset preprocessing, simulating low light environment;

[0009] S2. Low-light image is increased to obtain enhanced image I enhanced ;

[0010] S3. Enhanced Image I enhanced Calibrate to get the calibration image I calibrated ;

[0011] S4. Extract features from the image and output feature vectors; use the improved residual network model ResNet18 to extract features from the calibrated image;

[0012] S5. Map the feature vector into a three-dimensional output vector O through a fully connected layer;

[0013] S6. Apply a hyperbolic tangent transform to the first two elements of the three-dimensional output vector O to obtain an accurate predicted line of sight direction;

[0014] S7. The third element of the three-dimensional output vector O is transformed by a sigmoid function to obtain the uncertainty of the line of sight prediction;

[0015] S8. The MSELoss loss function is used to measure the error between the predicted result and the true value, and the network parameters are updated through back propagation.

[0016] Furthermore, in S1, in order to simulate the visual effects in different low-light environments, the Gaze360 dataset was preprocessed and a new Gaze360 dataset reflecting various low-light conditions was constructed;

[0017] Classify low-light environments, including: dark scenes, extremely dark scenes, low-light environments simulated using gamma correction, and simulated dark scenes with unknown light source locations;

[0018] The specific steps of S1 are as follows:

[0019] S101. For darker and extremely dark scenes, the brightness and contrast of the image are adjusted to increase the realism of night vision; specifically, the brightness adjustment is achieved by changing the range of dark and bright parts of the image; the contrast adjustment is achieved by adjusting the distribution range of colors in the image, so that the color display is more concentrated, thereby strengthening the distinction between dark and bright parts;

[0020] S102. For a low-light environment image set simulated using gamma correction, O=I (1 / G) ×255 to get the gamma-corrected output image O, where I is the input image and G is the gamma value;

[0021] S103. For a set of images of simulated dark scenes with unknown light source positions, the image is darkened and a local light source effect is added; first, the overall brightness of the image is significantly reduced by adjusting the brightness to enhance the performance of the dark part in the night environment; then, a gradient light source effect is introduced at random positions of the image and Gaussian blur is applied to simulate the illumination of a specific light source to ensure that the lighting effect is naturally integrated into the scene.

[0022] Furthermore, in step S2, the low-light image is enhanced by an enhancement network module; the enhancement network module is designed to extract multi-scale features from the input image and combine global context information to ultimately achieve detail enhancement and illumination balance; the enhancement network module consists of an input convolutional layer, an ALGCF module, multiple convolutional blocks, and an output convolutional layer;

[0023] The specific steps of S2 are as follows:

[0024] S201. Initial feature extraction: The input image I passes through the initial convolution layer and extracts the basic features:

[0025] F0 = ReLU(Conv0(I))

[0026] Conv0 is the initial convolution layer, which extracts the features of the input image through the convolution kernel; the ReLU activation function is used for nonlinear processing, so that the feature map F0 maintains the basic lighting and detail effects, providing reliable initial features for subsequent fusion and enhancement;

[0027] S202. Adaptive local-global fusion, i.e. ALGCF;

[0028] Extract detailed information of the eye area through the local feature extractor:

[0029] F local =Conv1(F0)

[0030] Among them, Conv1 is the convolution layer of the local feature extractor, which focuses on extracting eye details from the initial feature map; the local feature extractor captures detail features through the convolution kernel;

[0031] The global context information extractor is used to obtain the overall illumination and structure information of the face:

[0032] F global =In(Conv2(AAP(F0)),size=F local )

[0033] Among them, AAP is an adaptive average pooling layer that can extract illumination and structural information from the global context; Conv2 uses a 1×1 convolution kernel to generate a global feature map; then the global feature is adjusted to the same size as the local feature through the interpolation In operation for fusion;

[0034] Combine local and global features, generate fusion weights, and perform adaptive feature fusion through fusion weights:

[0035]

[0036] Among them, Concat combines local and global features into a fused feature map; Conv3 is the convolution layer of the gating mechanism, σ represents the Sigmoid function, and the fusion weight G is generated by the Sigmoid function; the fusion feature F ALGCF Adaptively fuse local and global features based on fusion weights G and 1-G to ensure that the facial feature map has global illumination consistency and maintains local details of the eyes;

[0037] S203. Fused feature map F ALGCF Further features are extracted through multiple convolution blocks:

[0038] F i+1 =F i +ReLU(BN(Conv i (F i )))

[0039] Conv i is the convolutional layer of each convolutional block, F i is the feature map of the i-th layer; BN, i.e. BatchNorm, performs feature normalization, and the ReLU activation function ensures nonlinear processing of features; through the residual connection (+) operation, each convolution block maintains the integrity of the input features while extracting features;

[0040] The output convolutional layer transforms the fused feature map into an enhanced image:

[0041] F output =σ(Conv4(F i+1 ))

[0042] Conv4 is the output convolution layer, which converts the feature map into the final enhanced image through convolution and normalizes the pixel values ​​to the range of 0 to 1 through the Sigmoid activation function;

[0043] S204. Finally, the enhanced feature map is added to the input image to obtain the final enhanced image:

[0044] I enhanced =Clamp(F output +I,0,1)

[0045] The Clamp function ensures that the pixel values ​​of the enhanced image are within a reasonable range. By enhancing and fusing the details of the original image, the resulting enhanced image has higher brightness and contrast.

[0046] Furthermore, in step S3, the enhanced image I enhanced After the initial convolutional layer, the calibration feature map is obtained:

[0047] F calib(0) =ReLU(BN(Conv5(I enhanced )))

[0048] Among them, Conv5 is the input convolution layer of the calibration network, which extracts the calibration feature map and normalizes it through BatchNorm; ReLU activation function is used for nonlinear processing; F calib(0) It represents the feature map obtained after the enhanced image is processed by the initial convolution layer, that is, the calibration feature map;

[0049] Calibration feature map F calib(0) After multiple convolution blocks, the lighting and details are adjusted:

[0050] F calib(i) =F calib(i-1) +ReLU(BN(Conv i (F calib(i-1) )))

[0051] Among them, F calib(i-1) It represents the output feature map of the i-1th layer, which is also the input of the i-th convolutional block, providing input data for the current layer and containing the accumulated feature information processed by the previous layers; F calib(i) is the output feature map of the i-th convolutional block, combined with the original input feature F calib(i-1)and new features processed by convolution, batch normalization, and ReLU; residual connections (+) help prevent the gradient vanishing problem in deep networks, while ensuring that enough original information is retained even in deep networks;

[0052] The final calibrated feature map is passed through the output convolution layer to generate a difference image:

[0053] ΔI=σ(Conv out (F calib_i ))

[0054] Among them, Conv out It is the output convolution layer, which generates the final difference image through convolution and ensures that the output value is within a reasonable range through the Sigmoid function;

[0055] Finally, the difference image is subtracted from the augmented input image to obtain the calibrated final image:

[0056] I calibrated =I enhanced -ΔI

[0057] The calibration module eliminates uneven illumination and artifacts in the enhanced image through subtraction operations, making the final calibrated image closer to the ideal state.

[0058] Furthermore, in step S4, the improved residual network model ResNet18 is used to extract features from the calibrated image, an attention mechanism module is introduced in the second and third stages, and a feature purification module is introduced in the fourth stage.

[0059] According to one aspect of the present invention, there is provided a storage medium, wherein instructions are stored in the storage medium. When a computer reads the instructions, the computer executes any one of the above-mentioned high-precision line of sight estimation methods in a low-light environment.

[0060] According to another aspect of the present invention, there is provided an electronic device, comprising a processor and the above-mentioned storage medium, wherein the processor executes instructions in the storage medium.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. This paper introduces low-light image enhancement technology, which focuses on improving the clarity of eye details in low-light environments through multi-scale feature fusion and global information extraction. The synergy of the enhancement and calibration modules significantly improves the visibility and quality of the image, providing high-quality input data for line of sight estimation, thereby enhancing the accuracy and reliability of the model.

[0063] 2. The present invention proposes a gaze estimation model that combines feature purification and attention mechanism. The feature purification module effectively extracts eye details through adaptive weights and regional attention mechanism, while the attention mechanism achieves more accurate gaze estimation by integrating spatial and channel weights and feature correlation analysis. After extensive testing, the angular errors of this method on the Gaze360 dataset in four low-light environments, Dark_comp, Dark_super, Dark_gamma, and Dark_light, are 12.75°, 13.27°, 12.43°, and 11.57°, respectively, which are better than the existing advanced network models.

[0064] 3. This invention not only opens up a new research direction of using low-light image enhancement technology to improve line of sight estimation in low-light environments, but also provides new ideas and solutions for line of sight estimation technology. In addition, this technology provides important technical support for application fields such as human-computer interaction in low-light environments and driver fatigue monitoring, broadening its application scope. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0066] Unless otherwise defined, technical or scientific terms used herein shall have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs.

[0067] Figure 1 This is a line of sight estimation model diagram in Embodiment 1 of the present invention;

[0068] Figure 2 This is a diagram of the FWF-Gaze ​​network framework in Example 1 of the present invention;

[0069] Figure 3 This is a diagram of an enhanced network model for low-light images in Example 1 of the present invention;

[0070] Figure 4 This is a calibration network model diagram for low-light images in Example 1 of the present invention;

[0071] Figure 5 is the attention model diagram in Example 1 of the present invention;

[0072] Figure 6 This is a feature purification network model diagram in Example 1 of the present invention;

[0073] Figure 7 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] like Figure 1 As shown in the figure, a high-precision line of sight estimation method in low-light environments is proposed. This method uses an improved residual network model ResNet18. By introducing attention mechanism modules in the second and third stages and feature purification modules in the fourth stage, the performance of the model in various low-light environments is significantly optimized. In addition, the introduced low-light image enhancement technology combines the enhancement module and the calibration module to effectively improve the visibility and quality of image details. In particular, the enhancement module specifically enhances the details of the eye area through multi-scale feature fusion and extraction of global context information, while the calibration module further refines the image to eliminate artifacts that may be introduced by the enhancement process.

[0076] Figure 2 The overall network framework of the FWF-Gaze ​​model proposed in this paper is demonstrated, which is specifically used for line of sight estimation in low-light environments. The image is processed by the image enhancement network and then passed to the line of sight estimation network. This end-to-end integrated approach enables the FWF-Gaze ​​model to show excellent performance in line of sight estimation problems in complex lighting environments.

[0077] In order to achieve the above requirements, the technical solution adopted by the present invention is as follows:

[0078] A high-precision line of sight estimation method in a low-light environment specifically comprises the following steps:

[0079] S1: Preprocess the Gaze360 dataset to simulate four low-light environments and prepare the data for subsequent image processing and analysis.

[0080] S2: The low-light image is input into the enhancement network and the enhanced image I is obtained after processing. enhanced , to improve image visibility and detail clarity in low-light conditions.

[0081] S3: Enhanced Image I enhanced After further processing by the calibration network, the final calibration image I is output calibrated , to remove noise and artifacts that may be introduced by the enhancement process.

[0082] S4: Use the improved residual network model ResNet18 to extract features from the calibrated image and output the feature vector.

[0083] S5: The feature vector is mapped into a three-dimensional output vector O through a fully connected layer, which contains the predicted line of sight direction (horizontal and vertical angles) and its predicted uncertainty (angle error).

[0084] S6: Apply a hyperbolic tangent transform to the first two elements of the three-dimensional output vector O to obtain the precise predicted viewing direction.

[0085] S7: The third element of the three-dimensional output vector O is transformed by the sigmoid function to obtain the uncertainty of the line of sight prediction.

[0086] S8: The MSELoss loss function is used to measure the error between the predicted result and the true value, and the network parameters are updated through back propagation to optimize the performance and accuracy of the model.

[0087] Preferably, in step S1, in order to simulate the visual effects in different low-light environments, the present invention performs four specific preprocessing techniques on the Gaze360 dataset. This includes: darker scenes

[0088] The present invention successfully constructs a new Gaze360 dataset reflecting various low-light conditions by using three methods: (Dark_comp), extremely dark scenes (Dark_super), low-light environments simulated using gamma correction (Dark_gamma), and simulated dark scenes with unknown light sources (Dark_light).

[0089] S101: For darker scenes (Dark_comp) and extremely dark scenes (Dark_super), the present invention adjusts the brightness and contrast of the image to increase the realism of night vision. Specifically, the brightness adjustment is achieved by changing the dark and bright intervals of the image; the contrast adjustment is achieved by adjusting the distribution range of colors in the image, making the color display more concentrated, thereby strengthening the distinction between dark and bright areas.

[0090] S102: For a low illumination environment (Dark_gamma) image set simulated by gamma correction, the present invention uses O=I (1 / G) ×255 to obtain the gamma-corrected output image O, where I is the input image and G is the gamma value. When the gamma value is greater than 1, the brightness of the image increases accordingly. Similarly, when the gamma value is less than 1, the image becomes darker. The closer the gamma value is to 0, the more difficult it is for the eye to recognize. Therefore, the present invention corrects the gamma value of the image and finally selects a gamma value of 0.7 to simulate a low-light environment.

[0091] S103: In order to further simulate low-light conditions in reality, such as museums or night driving scenes, the present invention develops specific image processing steps, including image darkening and adding local light source effects (Dark_light). First, the overall brightness of the image is significantly reduced by adjusting the brightness to enhance the performance of dark areas in night environments. Subsequently, by introducing a gradual light source effect at random positions in the image and applying Gaussian blur, the illumination of a specific light source, such as a street lamp, is simulated to ensure that the lighting effect blends naturally in the scene, thereby enhancing the realism and visual focus of the scene.

[0092] Preferably, in step S2, the enhancement network module aims to extract multi-scale features from the input image and combine the global context information to finally achieve detail enhancement and illumination balance. It consists of an input convolution layer, an ALGCF module, multiple convolution blocks and an output convolution layer. The image enhancement network model is as follows: Figure 3 shown.

[0093] The specific method is as follows:

[0094] S201: First, perform initial feature extraction. The input image I passes through the initial convolution layer to extract basic features:

[0095] F0 = ReLU(Conv0(I))

[0096] Conv0 is the initial convolution layer, which extracts the features of the input image through the convolution kernel. The ReLU activation function is used for nonlinear processing, so that the feature map F0 maintains the basic lighting and detail effects, providing reliable initial features for subsequent fusion and enhancement.

[0097] S202: Next, an adaptive local-global fusion (ALGCF) module is performed, which is responsible for fusing local and global information.

[0098] The local feature extractor is used to extract detailed information of the eye area:

[0099] F local =Conv1(F0)

[0100] Among them, Conv1 is the convolution layer of the local feature extractor, which focuses on extracting eye details from the initial feature map. The local feature extractor captures important detail features through the convolution kernel.

[0101] The global context information extractor is used to obtain the overall illumination and structure information of the face:

[0102] F global =In(Conv2(AAP(F0)),size=F local )

[0103] Among them, AAP is an adaptive average pooling layer that can extract illumination and structural information from the global context. Conv2 uses a 1×1 convolution kernel to generate a global feature map. The global feature is then adjusted to the same size as the local feature through the interpolation In operation for fusion.

[0104] Combine local and global features, generate fusion weights, and perform adaptive feature fusion through fusion weights:

[0105]

[0106] Among them, Concat combines local and global features into a fused feature map. Conv3 is the convolution layer of the gating mechanism. σ represents the Sigmoid function, which is the same in the following. The fusion weight G is generated by the Sigmoid function to ensure that the information from different sources is reasonably combined. Fusion feature F ALGCF The local and global features are adaptively fused according to the weights G and 1-G to ensure that the facial feature map has global illumination consistency and maintains local details of the eyes.

[0107] S203: Fused feature map F ALGCF After multiple rolls, further features are extracted:

[0108] F i+1 =F i +ReLU(BN(Conv i (F i )))

[0109] Conv i is the convolutional layer of each convolutional block, F i is the feature map of the i-th layer. BatchNorm performs feature normalization and the ReLU activation function ensures nonlinear processing of features. Through the residual connection (+) operation, each convolution block maintains the integrity of the input features while extracting features.

[0110] The output convolutional layer transforms the fused feature map into an enhanced image:

[0111] F output =σ(Conv4(F i+1 ))

[0112] Conv4 is the output convolution layer, which converts the feature map into the final enhanced image through convolution and normalizes the pixel values ​​to the range of 0 to 1 through the Sigmoid activation function.

[0113] S204: Finally, the enhanced feature map is added to the input image to obtain the final enhanced image:

[0114] I enhanced =Clamp(Foutput +I,0,1)

[0115] The Clamp function ensures that the pixel values ​​of the enhanced image are within a reasonable range. By enhancing and fusing the details of the original image, the resulting enhanced image has higher brightness and contrast.

[0116] Preferably, in step S3, after the calibration network processing, the calibration network model is as follows: Figure 4 As shown, the enhanced image I enhanced After the initial convolutional layer, the calibration feature map is obtained:

[0117] F calib(0) =ReLU(BN(Conv5(I enhanced )))

[0118] Among them, Conv5 is the input convolution layer of the calibration network, which extracts the calibration feature map and standardizes it through BatchNorm to ensure the stability and consistency of the feature map. The ReLU activation function is used for nonlinear processing to make the features more identifiable. calib(0) It represents the feature map obtained after the enhanced image is processed by the initial convolution layer. This is the first feature map in the calibration module and serves as the basis for subsequent convolution blocks.

[0119] Calibration feature map F calib-0 After multiple convolution blocks, the lighting and details are adjusted:

[0120] F calib(i) =F calib(i-1) +ReLU(BN(Conv i (F calib(i-1) )))

[0121] Among them, F calib(i-1) It represents the output feature map of the i-1th layer and is also the input of the i-th convolutional block. It provides input data for the current layer and contains the accumulated feature information processed by the previous layers. calib(i) is the output feature map of the i-th convolutional block, which combines the original input features F calib(i-1) and new features processed by convolution, batch normalization, and ReLU. Residual connections (+) help prevent the gradient vanishing problem in deep networks while ensuring that enough original information is preserved even in deep networks.

[0122] The final calibrated feature map is passed through the output convolution layer to generate a difference image:

[0123] ΔI=σ(Conv out (F calib_i ))

[0124] Among them, Convout It is the output convolution layer, which generates the final difference image through convolution and ensures that the output value is within a reasonable range through the Sigmoid function.

[0125] Finally, the difference image is subtracted from the augmented input image to obtain the calibrated final image:

[0126] I calibrated =I enhanced -ΔI

[0127] The calibration module eliminates uneven illumination and artifacts in the enhanced image through subtraction operations, making the final calibrated image closer to the ideal state.

[0128] Preferably, in step S4, an improved residual network model ResNet18 is used to extract features from the calibrated image, and the performance of the model in various low-light environments is significantly optimized by introducing an attention mechanism module in the second and third stages and a feature purification module in the fourth stage.

[0129] S401: Attention mechanism module:

[0130] Spatial attention focuses on enhancing specific areas of the image, allowing the model to pay more attention to key areas such as the eyes. Figure 5 As shown. The present invention uses a larger convolution kernel to capture a wider range of contextual information to emphasize the importance of the eye area:

[0131] S(x)=(Conv spatial (x)

[0132] Where S(x) represents a convolutional layer Conv spatial The spatial weight map obtained after (using a 7×7 convolution kernel and appropriate padding to keep the feature map size unchanged).

[0133] Channel attention evaluates the contribution of each channel to gaze estimation and optimizes the quality of the overall feature by strengthening the features of important channels and suppressing unimportant channels.

[0134] C(x)=σ(Conv ↑ (ReLU(Conv ↓ (GAP(x)))))

[0135] GAP(x) represents global average pooling of input x, compressing the spatial information of each channel into a single value. Then it is processed through two layers of convolution operations (first reducing the dimension and then increasing the dimension) and ReLU activation function, and finally the channel weight is generated through the Sigmoid function σ.

[0136] Next, feature correlation analysis is introduced, which not only integrates the attention of space and channel, but also optimizes the interaction between features and enhances the integration effect of features. The weighted features of space and channel attention are connected in the channel dimension to obtain feature F concat .

[0137] F concat =[x⊙S(x),x⊙C(x)]

[0138] Among them, S(x)·x and C(x)·x are the feature maps with spatial and channel weights applied, respectively.

[0139] Then, feature correlation analysis is performed on the merged features.

[0140] R(x)=σ(Conv ↑ (ReLU(Conv ↓ (F concat ))))

[0141] Conv ↓ It is a dimensionality reduction convolution layer, which helps the module focus on capturing the most critical feature correlation information by reducing the number of channels. The ReLU activation function is used here to increase nonlinearity, so that the model can capture more complex feature relationships. ↑ is an extended convolution layer that restores the number of channels to the original state and generates the final feature adjustment weights. These weights are adjusted to a reasonable range through the Sigmoid function to ensure that they can be used as an effective scaling factor to adjust the original input

[0142] Through feature correlation analysis, the model can dynamically evaluate and adjust the relationship between features, so that important features are highlighted and irrelevant or noisy features are suppressed. This sophisticated feature processing strategy significantly improves the performance of the model under complex lighting and low-light conditions, ensuring the accuracy and robustness of line of sight estimation.

[0143] Combining the above components, the enhanced feature F R(x) It can be obtained by following the steps below:

[0144] F R(x) = x·R(x)

[0145] By combining spatial and channel attention, as well as feature correlation analysis, the proposed attention mechanism can effectively identify and enhance key eye features for gaze estimation, especially in changing lighting and complex visual scenes.

[0146] S402: Feature purification module:

[0147] Using depth-separable convolution for basic feature extraction, this method effectively reduces the computational complexity while maintaining extraction efficiency:

[0148] F basic =ReLU(BN(Conv(x)))

[0149] Among them, Conv(x) represents group convolution, which is used to extract group features. Group convolution reduces the complexity of the model, which helps to keep parameters independent between different channels and reduce the number of parameters. basic Represents the feature map after preliminary processing.

[0150] The adaptive weight layer optimizes the representation of key features by learning the importance of each feature to adjust its contribution to the final output:

[0151] W adaptive =σ(Conv adaptive (F basic ))

[0152] Where W adaptive Represents the adaptive weights generated for the feature map, Conv adaptive It includes two-step convolution operations to adjust feature weights by reducing and expanding the number of channels.

[0153] The regional attention module focuses on the eye region and enhances important features by generating a region-specific attention mask:

[0154] M focus =σ(Conv focus (GAP(F basic )))

[0155] Among them, GAP focuses on global information and generates context for local features, M focus To focus on the mask, Conv focus Contains a convolution step that converts global information into an attention mask.

[0156] Finally, combined with the adaptive weight W adaptive and attention mask M focus , for detail enhancement and feature optimization:

[0157] F enhanced =(F basic ×M focus )×W adaptive

[0158] The combination of focus and weight adjustment effectively improves the expression of eye details and the quality of overall features. This module ensures that high-quality and high-precision eye features can be obtained in line of sight estimation by accurately adjusting the contribution and focus area of ​​each feature. Figure 6 shown.

[0159] S403: Residual Connection Module

[0160] In the ResNet18 baseline model, the input image is first processed through a 7×7 convolution layer, a BN layer, and a ReLU activation function, and then a 3×3 max pooling layer is applied. The network then goes through four stages, each of which passes through a residual block. We added a feature purification module and an attention mechanism module to the ResNet18 baseline model to increase the accuracy of line of sight estimation in low-light scenes. The feature extraction network structure after low-light processing is shown in the figure below. Figure 7 As shown, Res1, Res2, Res3, and Res4 represent four residual stages, and each stage has a similar structure.

[0161] After the Res2 and Res3 stages, attention mechanism modules are integrated. These modules optimize feature recognition in key visual areas through spatial and channel attention and feature correlation analysis. This helps the model to more accurately process and utilize key information from the eye, thereby achieving more accurate and stable gaze tracking in various low-light environments.

[0162] F attention =EyeFocusedSCAttentionModule(F input )

[0163] After the Res4 stage, a feature purification module is embedded. This module optimizes the model's perception of eye details through adaptive weights and regional attention mechanisms. This strategy is based on the importance of eye features in gaze estimation, especially in low light or complex lighting conditions.

[0164] F purify =EnhancedFeaturePurificationModule(F input )

[0165] By alternating feature purification and attention mechanisms at different stages, it is possible to optimize line of sight estimation requirements in various low-light scenarios while maintaining the advantages of deep networks.

[0166] Preferably, in steps S5-S8, in order to evaluate the prediction ability of the model, the present invention uses the mean square error MSELoss as a loss function to measure the difference between the prediction result and the actual fixation point. The loss function MSELoss of this module is expressed as:

[0167]

[0168] Among them, Lmse represents the mean square error loss, P i and Y i Represent the predicted value and true value of the i-th sample respectively. mse Optimize the model. By back-propagating the error, the parameters of the network can be iteratively updated, thereby gradually improving the performance of the model in the line of sight estimation task.

[0169] Embodiment 1:

[0170] Dataset,Gaze360 dataset is video data collected from 238 subjects in the real world. The dataset is large in scale, combining 3D gaze annotations, a wide range of gaze and head poses, various indoor and outdoor capture environments, and diversity of subjects. In the Gaze360 dataset, the present invention follows the predefined training-test split and only uses 84,902 frontal face images for training. The test set contains 16,031 images to comprehensively evaluate the performance of the model. The entire FWF-Gaze ​​network is trained using four preprocessed Gaze360 datasets (Dark_comp, Dark_super, Dark_gamma, Dark_light).

[0171] Environment: We used Windows environment and Pytorch framework to conduct experiments on NVIDIA GeForce RTX 3090 GPU. For both datasets, the model hyperparameters were set the same. The model was trained for 100 epochs, the batch size was 40, the learning rate was set to 0.0001, and the decay was set to 1. The decay step size was set to 5000.

[0172] Evaluation indicators: We use the mainstream evaluation indicator of gaze estimation, angle error, to compare the performance of other gaze estimation models. That is, the deviation angle between the predicted value and the true value of gaze estimation is expressed numerically. The smaller the indicator, the better the effect. Assuming the actual gaze direction is The estimated gaze direction is The angle error can then be calculated as:

[0173]

[0174] The comparison model uses advanced gaze estimation methods such as Gaze360, FullFace, RT-Gene, and Dilated-Net. The experimental settings of each method use the settings in the corresponding paper, including the model framework and hyperparameters, to reproduce their network performance.

[0175] The experimental results are shown in Table 1:

[0176] Table 1 Experimental results of the network proposed by this invention and other advanced networks

[0177]

[0178] It can be seen from the experimental data in Table 1 that, after experimental verification, the method of the present invention can effectively solve the problem of reduced accuracy of line of sight estimation in low-light environments, and has strong practical value.

[0179] Embodiment 2:

[0180] This embodiment will introduce an applicable scenario of the present invention:

[0181] Gaze estimation has a wide range of application scenarios, one of which is the intelligent interactive system. The FWF-Gaze ​​network in this paper can be used to detect driver fatigue. This application scenario helps to improve safety and convenience.

[0182] An important indicator of driver fatigue status is the driver's gaze state, and the FWF-Gaze ​​network model of the present invention is used to predict the gaze.

[0183] First, when the driver is driving the vehicle, the driver's facial image can be captured in real time through the camera.

[0184] Then, it is input into the FWF-Gaze ​​network model of the present invention to predict the driver's gaze state.

[0185] Finally, when the model predicts that the driver may be fatigued, the system can remind the driver to rest through sound or other means, or automatically switch to autonomous driving mode (if the vehicle supports it).

[0186] The examples described in the present invention are merely descriptions of the preferred implementation modes of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by engineers and technicians in this field should all fall within the protection scope of the present invention.

Claims

1. A high-precision line of sight estimation method in a low-light environment, characterized in that: The specific steps include: S1. Dataset preprocessing, simulating low light environment; S2. Low-light image enhancement to obtain enhanced image I enhanced ; S3. Enhanced Image I enhanced Calibrate to get the calibration image I calibrated ; S4. Extract features from the image and output feature vectors; Use the improved residual network model ResNet18 to extract features from the calibration image; S5. Map the feature vector into a three-dimensional output vector O through a fully connected layer; S6. Apply a hyperbolic tangent transform to the first two elements of the three-dimensional output vector O to obtain a predicted sight direction; S7. The third element of the three-dimensional output vector O is transformed by a sigmoid function to obtain the uncertainty of the line of sight prediction; S8. Use the MSELoss loss function to measure the error between the predicted result and the true value, and update the network parameters through back propagation; In step S2, the low-light image is enhanced by an enhancement network module; the specific steps of S2 are as follows: S201. Initial feature extraction: Input image I passes through the initial convolution layer to extract basic features: F0 = ReLU(Conv0(I)) Conv0 is the initial convolutional layer; S202. Adaptive local-global fusion, i.e. ALGCF; Extract detailed information of the eye area through the local feature extractor: F local =Conv1(F0) Among them, Conv1 is the convolution layer of the local feature extractor; The global context information extractor is used to obtain the overall illumination and structure information of the face: F global =In(Conv2(AAP(F0)),size=F local , Among them, AAP is an adaptive average pooling layer; Conv2 uses a 1×1 convolution kernel to generate a global feature map; then the global feature is adjusted to the same size as the local feature through the interpolation In operation; Combine local and global features, generate fusion weights, and perform adaptive feature fusion through fusion weights: Among them, Concat combines local and global features into a fused feature map; Conv3 is the convolution layer of the gating mechanism, σ represents the Sigmoid function, and the fusion weight G is generated by the Sigmoid function; the fusion feature F ALGCF Adaptively fuse local and global features according to fusion weights G and 1-G; S203. Fused feature map F ALGCF Further features are extracted through multiple convolution blocks: F i+1 =F i +ReLU(BN(Conv i (F i ))) Conv i is the convolutional layer of each convolutional block, F i is the feature map of the i-th layer; the output convolution layer converts the fused feature map into an enhanced image: F output =σ(Conv4(F i+1 )) Conv4 is the output convolution layer, which converts the feature map into an enhanced image through convolution and normalizes the pixel values ​​to the range of 0-1 through the Sigmoid activation function; S204. Add the enhanced feature map to the input image to obtain the final enhanced image: I enhanced =Clamp(F output +I,0,1) The Clamp function ensures that the pixel values ​​of the augmented image are within the range.

2. The method according to claim 1, characterized in that In S1, the Gaze360 dataset is preprocessed to construct a new Gaze360 dataset reflecting various low-light conditions; Classify low-light environments, including: dark scenes, extremely dark scenes, low-light environments simulated using gamma correction, and simulated dark scenes with unknown light source locations; The specific steps of S1 are as follows: S101. For darker scenes and extremely dark scenes, adjust the brightness and contrast of the image; S102. For a low-light environment image set simulated using gamma correction, O=I (1 / G) ×255 to get the gamma-corrected output image O, where I is the input image and G is the gamma value; S103. For a set of images of simulated dark scenes with unknown light source positions, the image is darkened and a local light source effect is added; first, the overall brightness of the image is significantly reduced by adjusting the brightness to enhance the performance of the dark part in the night environment; then, a gradient light source effect is introduced at random positions of the image and Gaussian blur is applied to simulate the illumination of a specific light source to ensure that the lighting effect is naturally integrated into the scene.

3. The method according to claim 1, characterized in that In step S3, the enhanced image I enhancea After the initial convolutional layer, the calibration feature map is obtained: F calib(0) =ReLU(BN(Conv5(I enhanced ))) Among them, Conv5 is the input convolution layer of the calibration network; F calib (0) represents the feature map obtained after the enhanced image is processed by the initial convolution layer, i.e., the calibration feature map; Calibration feature map F calib(0) After multiple convolution blocks, the lighting and details are adjusted: F calib(i) =F calib(i-1) +ReLU(BN(Conv i (F calib(i-1) ))) Among them, F calib(i-1) represents the output feature map of the i-1th layer; F calib(i) is the output feature map of the i-th convolutional block; The final calibrated feature map is passed through the output convolution layer to generate a difference image: ΔI=σ(Conv out (F calib_i )) Among them, Conv out is the output convolutional layer; I calibrated =I enhanced -ΔI.

4. The method according to claim 3, characterized in that In step S4, the improved residual network model ResNet18 is used to extract features of the calibration image, an attention mechanism module is introduced in the second and third stages, and a feature purification module is introduced in the fourth stage.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the high-precision sight line estimation method in a low-light environment as described in any one of claims 1 to 4 are implemented.

6. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the high-precision sight line estimation method in a low-light environment are implemented as described in any one of claims 1 to 4.

Images