A monocular vehicle-mounted line-of-sight direction and gaze region joint estimation method based on geometric normalization feature fusion

Abstract

This invention discloses a method for joint estimation of gaze direction and gaze region in a monocular vehicle-mounted camera based on geometrically normalized feature fusion. The method includes the following steps: acquiring a driver's face image captured by a monocular vehicle-mounted camera; obtaining the original representation and the geometrically normalized representation; performing multi-scale feature extraction on both the original and geometrically normalized representations to obtain original multi-scale features and normalized multi-scale features; performing feature fusion on the original and normalized multi-scale features to obtain a fused feature representation; performing joint estimation based on the fused feature representation to output the driver's gaze direction estimation result and gaze region classification result; and introducing geometric consistency constraints to correct or optimize the driver's gaze direction estimation result and gaze region classification result to obtain the final output result. This invention improves the accuracy and stability of gaze direction and gaze region classification in monocular vehicle-mounted scenes, and has strong engineering practical value and promising application prospects.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving and computer vision technology, and specifically relates to a method for joint estimation of monocular vehicle gaze direction and gaze region based on geometric normalization feature fusion. Background Technology

[0002] With the development of intelligent cockpits and advanced driver assistance systems (ADAS), driver monitoring systems (DMS) have gradually become an important component for improving driving safety. Among these systems, gaze direction and gaze area classification can be used to identify risky states such as driver distraction, fatigue, and gaze deviation from the road, and support human-machine interaction and safety policy triggering. In engineering applications, due to limitations in cost, installation space, and mass production complexity, monocular automotive cameras remain the most common perception configuration. Therefore, achieving stable and accurate gaze estimation under monocular conditions is of great significance.

[0003] However, the in-vehicle environment is significantly complex: the driver's head posture changes frequently, and the relative distance and viewing angle between the person and the camera fluctuate with seat position and driving behavior; in-vehicle lighting is subject to strong reflections, shadows, and day / night variations; simultaneously, the eye region is small in scale and has limited texture details in the image, and is easily affected by glasses, occlusions, and motion blur. These factors lead to significant changes in the geometry of facial and eye imaging, causing gaze estimation methods that rely solely on image appearance features to become less accurate and less predictive under cross-pose, cross-distance, and cross-lighting conditions.

[0004] In existing technologies, one type of method uses face / eye images as input and regresses the gaze direction or classifies the gaze region through deep networks; another type of method introduces head pose estimation, 3D model fitting, or multi-sensor configuration to enhance geometric constraints. However, the former often struggles to explicitly offset the distribution shift caused by changes in imaging geometry, resulting in insufficient robustness in real-world in-vehicle scenarios; the latter may rely on additional hardware, complex calibration, or strict geometric priors, increasing deployment costs and system complexity. Furthermore, the gaze direction and gaze region are usually intrinsically related; if they are modeled independently, the consistency constraint between them is easily overlooked, thus affecting the overall reliability of the joint task.

[0005] Therefore, there is an urgent need for a gaze estimation method for vehicle-mounted monocular scenarios. Without increasing hardware costs, this method should enhance the adaptability to changes in pose, distance, and illumination through reasonable geometric normalization and feature fusion mechanisms, and achieve joint modeling and consistency optimization of the gaze direction and gaze area to improve accuracy and stability in practical applications. Summary of the Invention

[0006] The purpose of this invention is to provide a method for joint estimation of gaze direction and gaze area in monocular vehicle-mounted systems based on geometrically normalized feature fusion, in order to solve the technical problems in the prior art where gaze estimation in monocular vehicle-mounted scenarios is easily affected by head posture, viewing distance changes and imaging distortion, and where gaze direction and gaze area lack joint optimization.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] 1. A method for joint estimation of gaze direction and gaze region in a monocular vehicle based on geometrically normalized feature fusion, characterized by the following steps:

[0009] Step S1: Acquire the driver's face image captured by the monocular vehicle camera;

[0010] Step S2: Based on the driver's face image obtained in step S1, obtain the original representation and the geometrically normalized representation; the original representation is used to retain the original imaging appearance information, and the geometrically normalized representation is used to align the driver's face image to a preset normalized imaging geometry by performing a geometric transformation on the image.

[0011] Step S3: Perform multi-scale feature extraction on the original representation and the geometrically normalized representation obtained in step S2 to obtain the original multi-scale features and the normalized multi-scale features.

[0012] Step S4: Perform feature fusion on the original multi-scale features and normalized multi-scale features obtained in step S3 to obtain a fused feature representation; the feature fusion is used to achieve cross-representation information interaction and complementary enhancement.

[0013] Step S5: Perform joint estimation based on the fusion feature representation obtained in step S4, and output the driver's gaze direction estimation result and the gaze region classification result; the gaze direction estimation result is a continuous parameter characterizing the gaze direction, and the gaze region classification result is a discrete prediction characterizing the gaze region category.

[0014] Step S6: During the model training phase, a geometric consistency constraint is introduced to jointly optimize the driver's gaze direction estimation result and the gaze region classification result. The model parameters are updated to obtain the final output result. The geometric consistency constraint is used to constrain the prediction under the original representation and the geometrically normalized representation to satisfy the preset geometric relationship.

[0015] Furthermore, step S1 includes the following sub-steps:

[0016] Step S101: In the vehicle scene, image frames or video frame sequences containing the driver's face are acquired using a monocular camera.

[0017] Step S102: Perform face detection on the image frames or video frame sequence acquired in step S101 to obtain the driver's face region, and crop and align the driver's face region to obtain the driver's face image. ;

[0018] Step S103: Process the face image obtained in step S102. Perform at least one preprocessing operation, including any one or a combination of size normalization, pixel normalization, noise reduction, brightness or contrast correction, as input for subsequent construction of the original representation and the geometrically normalized representation.

[0019] Furthermore, step S2 includes the following sub-steps:

[0020] Step S201, based on the driver's face image obtained in step S1 Acquire facial key points and determine the center position of the face to establish the original camera coordinate system. and virtual camera coordinates Take the direction vector from the original camera pointing to the center of the face as... And set the optical axis direction of the virtual camera coordinate system to ;

[0021] Step S202, select the direction relative to the optical axis. Non-collinear reference vectors Construct an orthogonal basis and form a rotation matrix. :

[0022]

[0023] Step S203: Set the preset standard sight distance Construct the scaling transformation matrix:

[0024]

[0025] And based on the original camera intrinsic parameter matrix With virtual camera intrinsic parameter matrix Construct the geometric transformation matrix:

[0026] ;

[0027] Step S204, using the geometric transformation matrix For the face image Perform a perspective transformation to obtain a geometrically normalized representation. :

[0028]

[0029] in, This is a perspective transformation operation used to align the original face image to a preset normalized imaging geometry.

[0030] Furthermore, step S3 includes the following sub-steps:

[0031] Step S301, extract the face images respectively As the original representation and the geometrically normalized representation Input homogeneous or heterogeneous feature extraction networks Features are extracted at multiple scales to obtain the original multi-scale feature set and the normalized multi-scale feature set:

[0032]

[0033] in, Let be a set of scales, and each scale feature satisfies ;

[0034] Step S302: Perform scale alignment operation on features of different scales so that each scale feature meets the preset alignment requirements in spatial resolution and / or channel dimension. The scale alignment operation includes upsampling, downsampling, convolutional mapping or a combination thereof.

[0035] Step S303: The aligned multi-scale features are used as input for subsequent cross-representation feature fusion to preserve both local detail information and global context information.

[0036] Furthermore, step S4 includes the following sub-steps:

[0037] Step S401, for the scale set any scale in The original multi-scale features With normalized multiscale features Feature alignment and dimensionality unification are performed to obtain feature pairs for fusion. , ;

[0038] Step S402, for the feature pair , Perform cross-representation information interaction to obtain separately enhanced features. and The cross-representation information interaction includes attention interaction, relevance matching, gating fusion, or a combination thereof;

[0039] Step S403: Adaptively weight the enhanced features based on channel recalibration to obtain channel-weighted features. and The channel weights are generated from global statistics.

[0040] Step S404: Perform cross-scale aggregation on the channel-weighted features at each scale to obtain the fused feature representation. The cross-scale aggregation includes splicing, weighted summation, pyramid aggregation, or a combination thereof.

[0041] 6. Furthermore, in step S402, the attention interaction satisfies the following relationship:

[0042] Generate shared query features:

[0043]

[0044] in, for and The resulting fused input representation; generating the first key-value pairs respectively. Second key-value pair ;

[0045] The multi-head attention computation form satisfies:

[0046]

[0047] in, , , , , For the projection matrix, For feature dimension, For querying the matrix, The key matrix, Value matrix, superscript This represents the matrix transpose operation.

[0048] Furthermore, in step S403, the channel weights satisfy:

[0049]

[0050] in, Indicates global average pooling. This represents a multilayer perceptron. This represents the activation function. This indicates weighted by channel. This is a feature representation of the input to the channel recalibration step. The calculated channel weight vector, These are the features after channel weighting.

[0051] Furthermore, in step S404, the feature representation is fused. satisfy:

[0052]

[0053] in, For cross-scale aggregation operators, and Representing the scale respectively Enhanced original features and normalized features It is a set of scales for multi-scale features; thus, a fusion feature representation that simultaneously contains appearance detail information and geometric stability information is obtained.

[0054] Furthermore, step S5 includes the following sub-steps:

[0055] Step S501, represent the fused feature representation obtained in step S4. Input the gaze direction estimation branch, and output continuous parameters of the gaze direction through regression. The continuous parameter is used to characterize the driver's line of sight direction;

[0056] Step S502, represent the fusion features Input the gaze region classification branch, and output the gaze region category probability vector according to the classification method. And based on this, the gaze region category is obtained. ;

[0057] Step S503, the continuous parameters of the line of sight direction are... With the gaze region category As a joint estimation output, the gaze direction continuity parameter shares the fused feature representation with the gaze region category. The characteristic expression of.

[0058] Furthermore, in step S501, the line-of-sight direction estimation branch satisfies the following mapping relationship:

[0059]

[0060] in, Returning the line of sight to the network;

[0061] In step S502, the gaze region classification branch satisfies the following relationship:

[0062]

[0063] in, For gaze region classification networks, Let be the predicted probability of the j-th gaze region category.

[0064] Furthermore, step S6 includes the following sub-steps:

[0065] Step S601: Based on the geometric transformation relationship obtained when constructing the geometric normalization representation in step S2, establish a preset geometric consistency relationship between the original representation and the prediction result under the geometric normalization representation, and introduce the preset geometric consistency relationship as a constraint term into the joint estimation process;

[0066] Step S602: After mapping the gaze direction prediction corresponding to the original representation and the gaze direction prediction corresponding to the geometrically normalized representation to the same coordinate system, calculate the consistency error, and construct the geometric consistency loss based on the consistency error. ;

[0067] Step S603: Construct the line-of-sight regression loss. Compared with gaze region classification loss The line-of-sight regression loss Used to measure and predict line of sight direction Compared to the actual line of sight The difference in the gaze region classification loss Cross-entropy loss is used;

[0068] Step S604: During the model training or optimization phase, a loss including gaze direction regression is used. gaze area classification loss and geometric consistency loss The joint objective function updates the model parameters, and the joint objective function satisfies:

[0069]

[0070] in, The weighting coefficients are used to obtain the final model by minimizing the joint objective function, and the gaze direction estimation result and gaze region classification result optimized by geometric consistency constraints are output.

[0071] Furthermore, in step S602, the geometric consistency loss is:

[0072]

[0073] in, and These are the direction vectors predicted from the line-of-sight directions under the two different representations;

[0074] In step S603, the gaze direction regression loss satisfy:

[0075]

[0076] in, This is a function that maps the line-of-sight direction parameter to a three-dimensional direction vector;

[0077] The gaze region classification loss satisfy:

[0078]

[0079] in, For the number of gaze region categories, For the true label of the gaze area, For the predicted first Class probability.

[0080] Beneficial effects:

[0081] This invention constructs a standardized representation through geometric normalization, which explicitly weakens the distribution shift caused by changes in head posture, viewing distance, and imaging distortion, thereby improving the robustness of monocular vehicle-mounted scenarios.

[0082] This invention improves the ability to represent complex lighting, occlusion and small-scale eye details by fusing cross-representation features of the original representation and geometrically normalized representation, while retaining appearance detail information and geometric stability information.

[0083] This invention achieves collaborative optimization of the two tasks by jointly modeling the gaze direction and the gaze area and introducing geometric consistency constraints, thereby improving the accuracy and stability of the joint estimation.

[0084] This invention is applicable to engineering applications such as vehicle driver monitoring without increasing the cost of additional sensors and hardware, and has good deployment feasibility and promotion value. Attached Figure Description

[0085] Figure 1 This is a flowchart of the method for joint estimation of monocular vehicle-mounted gaze direction and gaze region based on geometric normalization feature fusion proposed in this invention;

[0086] Figure 2 This is a flowchart of the algorithm for the joint estimation method of monocular vehicle-mounted gaze direction and gaze region based on geometric normalization feature fusion proposed in this invention.

[0087] Figure 3 This is a schematic diagram of the internal structure of the Attention Fusion Module in the monocular vehicle-mounted gaze direction and gaze region joint estimation method based on geometric normalization feature fusion proposed in this invention. Detailed Implementation

[0088] The technical solution of the present invention will be further described below with reference to the accompanying drawings. It should be understood that this embodiment is only used to explain the present invention and is not intended to limit the scope of protection of the present invention. Various equivalent substitutions or modifications made by those skilled in the art without departing from the spirit and substance of the present invention should fall within the scope of protection of the present invention.

[0089] These and other aspects of the embodiments of the invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the invention; however, it should be understood that the scope of the embodiments of the invention is not limited thereto. Rather, the embodiments of the invention include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0090] Please see Figure 1 and Figure 2 ,in Figure 1 This is a flowchart of the method for joint estimation of monocular vehicle-mounted gaze direction and gaze region based on geometric normalization feature fusion proposed in this invention; Figure 2 This is a flowchart of the algorithm of the method of the present invention. From... Figure 2 As can be seen from the above, the present invention mainly comprises three parts: geometric normalization representation construction, cross-representation attention fusion, and joint prediction and consistency optimization. Geometric normalization representation construction is used to align the original face image to a normal geometry, thereby reducing the influence of pose and distance changes. Cross-representation attention fusion is used to realize the information interaction and complementary enhancement between the original representation features and the normalized representation features. Joint prediction and consistency optimization is used to simultaneously output the gaze direction and gaze area results, and improve stability through geometric consistency constraints.

[0091] This invention provides a method for joint estimation of gaze direction and gaze region in monocular vehicle-mounted cameras based on geometrically normalized feature fusion. The specific process is as follows: Figure 1 As shown, it includes the following steps:

[0092] Step S1: Acquire the driver's face image captured by the monocular vehicle camera;

[0093] Step S1 specifically includes:

[0094] Step S101: In the vehicle-mounted scenario, a monocular camera captures a sequence of image frames or video frames containing the driver's face, and records the sequence as follows: The original image of the frame is ;

[0095] Step S102: Perform face detection and key point localization on the original image to obtain the bounding box of the driver's face region. and key point set ;

[0096] Step S103, based on the bounding box The original image is cropped and aligned to obtain a face image. and to Preprocessing such as size normalization and pixel normalization is performed to ensure that the image meets the input size requirements of subsequent networks. For ease of description, time superscripts are omitted below, and the face image is denoted as... .

[0097] Step S2: Construct the original representation and the geometrically normalized representation;

[0098] The purpose of step S2 is to construct two complementary representations under the same monocular input, namely the original representation. Geometric normalization representation .in Retain accurate appearance information. Geometric transformations are used to align to normal geometry to mitigate the effects of pose, distance, and imaging distortion.

[0099] Step S2 specifically includes:

[0100] Step S201, based on the set of facial key points obtained in step S1 Estimate the center position of the face And construct a direction vector from the camera to the center of the face. The optical axis direction of the virtual standard coordinate system is set as follows:

[0101]

[0102] Step S202, select the direction relative to the optical axis. Non-collinear reference vectors Construct an orthogonal basis and form a rotation matrix.

[0103]

[0104] Step S203: Set the preset standard sight distance Construct the scaling transformation matrix

[0105]

[0106] Step S204, set the original camera intrinsic parameter matrix With virtual camera intrinsic parameter matrix Constructing the geometric transformation matrix

[0107] ;

[0108] Step S205, using the geometric transformation matrix For the face image Perform a perspective transformation to obtain a geometrically normalized representation. :

[0109]

[0110] in, The perspective transformation operation obtains the original representation through the above process. Geometric normalization representation .

[0111] Step S3: Perform multi-scale feature extraction respectively;

[0112] The purpose of step S3 is to extract multi-scale features from the original representation and the normalized representation respectively, so as to preserve local texture details and global structural information at the same time, and provide a basis for subsequent fusion.

[0113] Step S3 specifically includes:

[0114] Step S301, the original characterization Input Feature Extraction Network This yields the original feature set at multiple scales:

[0115]

[0116] Step S302, geometric normalization representation Input homogeneous or heterogeneous feature extraction networks Features are extracted at multiple scales to obtain the original multi-scale feature set and the normalized multi-scale feature set:

[0117]

[0118] in, Let be a set of scales, and each scale feature satisfies .

[0119] Step S303: Perform scale alignment operations on features of different scales to ensure that each scale feature meets preset alignment requirements in spatial resolution and / or channel dimension. The scale alignment operations include upsampling, downsampling, convolutional mapping, or combinations thereof, thereby obtaining aligned features for fusion. .

[0120] Step S4: Cross-representation feature fusion to obtain fused feature representation;

[0121] The purpose of step S4 is to utilize the attention fusion module ( Figure 2 and Figure 3 The Attention FusionModule shown enables information interaction and complementary enhancement between the original multi-scale features and the normalized multi-scale features, resulting in a unified fused feature representation.

[0122] Step S4 specifically includes:

[0123] Step S401 (feature concatenation and shared query generation) is for the scale set. any scale in The original features after obtaining the aligned features Geometric normalization features The token is obtained by flattening and serializing the token in the spatial dimension. and To splice the two together (e.g.) Figure 3 In (symbols), constructing fusion input tokens :

[0124]

[0125] in This indicates a concatenation operation (or an equivalent addition / stacking method).

[0126] Merge input token A shared query vector for cross-branch interaction is generated through linear mapping. :

[0127]

[0128] in, This is the projection matrix.

[0129] Step S402 (Independent Key-Value Pair Extraction and Cross-Representation Interaction): While generating the shared query, the original feature branch and the normalized feature branch are independently mapped to generate corresponding keys and values. That is, from... The mapping yields the first key-value pair. ,from The mapping yields the first key-value pair. The above shared query features The key-value pairs from both branches are fed into a multi-head attention mechanism. The technical advantage of this design lies in using the same query feature to guide the calculation of attention weights, forcing the network to simultaneously align and fuse complementary information at the same semantic anchor point (such as eye geometry). Its calculation form is as follows:

[0130]

[0131] in, , , , , For the projection matrix, For feature dimension, For querying the matrix, The key matrix, Value matrix, superscript This represents the matrix transpose operation.

[0132] This results in an enhanced token representation (which may include residual joins):

[0133]

[0134] In step S403 (Transformer Decoding and Semantic Alignment), the features after multi-head attention interaction are fed into the corresponding first decoder (Transformer Decoder 1) and second decoder (Transformer Decoder 2). The decoders are used to further capture long-range contextual dependencies and restore the serialized features to a feature map form with spatial dimensions.

[0135] Step S404 (Channel Adaptive Recalibration and Residual Output): The decoded feature map is input to the Channel Attention module. This module calculates the weights of each channel using global statistical information and performs fine-grained feature recalibration. The mechanism is as follows:

[0136]

[0137] in Indicates global average pooling. This represents a multilayer perceptron. For activation function, This indicates weighted by channel. This is a feature representation of the input to the channel recalibration step. The calculated channel weight vector, This is the feature after channel weighting. When the original representation is affected by glasses, masks, or localized highlights, this module can adaptively suppress the response of damaged texture channels and amplify the stable geometric cue weights in the normalized representation. The final output achieves bidirectional enhanced fused features. .

[0138] Step S405: Perform cross-scale aggregation on the fused enhancement features at each scale to obtain the fused feature representation:

[0139]

[0140] in, For cross-scale aggregation operators, This is a set of scales for multi-scale features. This yields a fused feature representation that simultaneously includes appearance detail information and geometric stability information, which can be used for subsequent joint estimation of gaze direction and gaze region.

[0141] Step S5: Jointly estimate the output gaze direction and gaze area;

[0142] The purpose of step S5 is to: based on the fused feature representation Simultaneously outputting the gaze direction estimation result and the gaze region classification result, achieving collaborative modeling.

[0143] Step S5: Jointly estimate the output gaze direction and gaze area;

[0144] The purpose of step S5 is to: based on the fused feature representation Simultaneously outputting gaze direction estimation results and gaze region classification results, achieving collaborative modeling. Specifically, this includes:

[0145] Step S501, fuse the feature representation Input gaze direction estimation branch Output continuous parameters of the line of sight direction through regression. :

[0146]

[0147] Step S502, fuse the feature representation Input gaze region classification branch Output the probability of the j-th class in the gaze region. And obtain the gaze region category :

[0148]

[0149] Step S503, continuously adjust the viewing direction parameters. With gaze area category As a joint estimation output, the two are shared. Feature representation to achieve collaborative modeling.

[0150] Step S6: Introduce geometric consistency constraints to perform consistency optimization output.

[0151] The purpose of step S6 is to apply geometric consistency constraints to the joint estimation results using the geometric transformation relationship in step S2, thereby improving the prediction stability and robustness.

[0152] Specifically, the implementation process of step S6 includes the following steps:

[0153] Step S601: Predict the direction of gaze. The mapping is expressed as a three-dimensional unit direction vector, and the mapping function is denoted as... And based on the preset geometric relationship between the original representation and the geometrically normalized representation, a consistency constraint is constructed.

[0154] Step S602, construct geometric consistency loss Preferably, the consistency loss adopts cosine consistency:

[0155]

[0156] in, and The direction vector representations obtained under the two characterizations;

[0157] Step S603: Construct the line-of-sight regression loss. Compared with gaze region classification loss The gaze direction regression loss Used to measure and predict line of sight direction Compared to the actual line of sight The difference, preferably,

[0158] The line-of-sight regression loss is expressed in the form of angular error:

[0159]

[0160] in The gaze direction parameter is a function that maps to a three-dimensional direction vector; the gaze region classification loss is... Using cross-entropy loss:

[0161]

[0162] in, For the number of gaze region categories, For the true label of the gaze area, For the predicted first Class probability;

[0163] Step S604: During the model training or optimization phase, a loss including gaze direction regression is used. gaze area classification loss and geometric consistency loss The joint objective function updates the model parameters, and the joint objective function satisfies:

[0164]

[0165] in, These are the weighting coefficients; by minimizing the joint objective function The final model parameters are obtained, and the gaze direction and gaze area results are output during the inference phase with consistency constraints.

[0166] The proposed method for joint estimation of gaze direction and gaze region in monocular vehicle-mounted systems based on geometrically normalized feature fusion constructs a standardized representation through geometric normalization to weaken the influence of pose and distance changes. It achieves cross-representation information interaction and obtains fused feature representation through an attention fusion module, and introduces geometric consistency constraints in the joint estimation process to improve prediction stability. Thus, without increasing additional hardware costs, it improves the accuracy and robustness of joint estimation of gaze direction and gaze region in monocular vehicle-mounted scenarios, and has good engineering application value.

[0167] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for joint estimation of gaze direction and gaze region in monocular vehicle-mounted cameras based on geometrically normalized feature fusion, characterized in that: Includes the following steps: Step S1: Acquire the driver's face image captured by the monocular vehicle camera; Step S2: Based on the driver's face image obtained in step S1, obtain the original representation and the geometrically normalized representation; the original representation is used to retain the original imaging appearance information, and the geometrically normalized representation is used to align the driver's face image to a preset normalized imaging geometry by performing a geometric transformation on the image. Step S3: Perform multi-scale feature extraction on the original representation and the geometrically normalized representation obtained in step S2 respectively to obtain the original multi-scale features and the normalized multi-scale features. Step S4: Perform feature fusion on the original multi-scale features and normalized multi-scale features obtained in step S3 to obtain the fused feature representation; The feature fusion is used to achieve cross-representation information interaction and complementary enhancement; Step S5: Perform joint estimation based on the fusion feature representation obtained in step S4, and output the driver's gaze direction estimation result and the gaze region classification result; the gaze direction estimation result is a continuous parameter characterizing the gaze direction, and the gaze region classification result is a discrete prediction characterizing the gaze region category. Step S6: During the model training phase, a geometric consistency constraint is introduced to jointly optimize the driver's gaze direction estimation result and the gaze region classification result. The model parameters are updated to obtain the final output result. The geometric consistency constraint is used to constrain the prediction under the original representation and the geometrically normalized representation to satisfy the preset geometric relationship.

2. The method according to claim 1, characterized in that: Step S1 includes the following sub-steps: Step S101: In the vehicle scene, image frames or video frame sequences containing the driver's face are acquired using a monocular camera. Step S102: Perform face detection on the image frames or video frame sequence acquired in step S101 to obtain the driver's face region, and crop and align the driver's face region to obtain the driver's face image. ; Step S103: Process the face image obtained in step S102. Perform at least one preprocessing operation, including any one or a combination of size normalization, pixel normalization, noise reduction, brightness or contrast correction, as input for subsequent construction of the original representation and the geometrically normalized representation.

3. The method according to claim 2, characterized in that: Step S2 includes the following sub-steps: Step S201, based on the driver's face image obtained in step S1 Acquire facial key points and determine the center position of the face to establish the original camera coordinate system. and virtual camera coordinates Take the direction vector from the original camera pointing to the center of the face as... And set the optical axis direction of the virtual camera coordinate system to ; Step S202, select the direction relative to the optical axis. Non-collinear reference vectors Construct an orthogonal basis and form a rotation matrix. : Step S203: Set the preset standard sight distance Construct the scaling transformation matrix: And based on the original camera intrinsic parameter matrix With virtual camera intrinsic parameter matrix Construct the geometric transformation matrix: ； Step S204, using the geometric transformation matrix For the face image Perform a perspective transformation to obtain a geometrically normalized representation. : in, This is a perspective transformation operation used to align the original face image to a preset normalized imaging geometry.

4. The method according to claim 3, characterized in that: Step S3 includes the following sub-steps: Step S301, extract the face images respectively As the original representation and the geometrically normalized representation Input homogeneous or heterogeneous feature extraction networks Features are extracted at multiple scales to obtain the original multi-scale feature set and the normalized multi-scale feature set: in, Let be a set of scales, and each scale feature satisfies ; Step S302: Perform scale alignment operation on features of different scales so that each scale feature meets the preset alignment requirements in spatial resolution and / or channel dimension. The scale alignment operation includes upsampling, downsampling, convolutional mapping or a combination thereof. Step S303: The aligned multi-scale features are used as input for subsequent cross-representation feature fusion to simultaneously preserve local detail information and global context information.

5. The method according to claim 4, characterized in that: Step S4 includes the following sub-steps: Step S401, for the scale set any scale in The original multi-scale features With normalized multiscale features Feature alignment and dimensionality unification are performed to obtain feature pairs for fusion. , ; Step S402, for the feature pair , Perform cross-representation information interaction to obtain separately enhanced features. and The cross-representation information interaction includes attention interaction, relevance matching, gating fusion, or a combination thereof; Step S403: Adaptively weight the enhanced features based on channel recalibration to obtain channel-weighted features. and The channel weights are generated from global statistics. Step S404: Perform cross-scale aggregation on the channel-weighted features at each scale to obtain the fused feature representation. The cross-scale aggregation includes splicing, weighted summation, pyramid aggregation, or a combination thereof.

6. The method according to claim 5, characterized in that: In step S402, the attention interaction satisfies the following relationship: Generate shared query features: in, for and The resulting fused input representation; generating the first key-value pairs respectively. Second key-value pair ; The multi-head attention computation form satisfies: in, , , , , For the projection matrix, For feature dimension, For querying the matrix, The key matrix, Value matrix, superscript This represents the matrix transpose operation; In step S403, the channel weights satisfy: in, Indicates global average pooling. This represents a multilayer perceptron. This represents the activation function. This indicates weighted by channel. This is a feature representation of the input to the channel recalibration step. The calculated channel weight vector, These are the features after channel weighting; In step S404, the feature representation is fused. satisfy: in, For cross-scale aggregation operators, and Representing the scale respectively Enhanced original features and normalized features It is a set of scales for multi-scale features; thus, a fusion feature representation that simultaneously contains appearance detail information and geometric stability information is obtained.

7. The method according to claim 6, characterized in that: Step S5 includes the following sub-steps: Step S501, represent the fused feature representation obtained in step S4. Input the gaze direction estimation branch, and output continuous parameters of the gaze direction through regression. The continuous parameter is used to characterize the driver's line of sight direction; Step S502, represent the fused features Input the gaze region classification branch, and output the gaze region category probability vector according to the classification method. And based on this, the gaze region category is obtained. ; Step S503, the continuous parameters of the line of sight direction are... With the gaze region category As a joint estimation output, the gaze direction continuity parameter shares the fused feature representation with the gaze region category. The characteristic expression of.

8. The method according to claim 7, characterized in that: In step S501, the line-of-sight direction estimation branch satisfies the following mapping relationship: in, Returning the line of sight to the network; In step S502, the gaze region classification branch satisfies the following relationship: in, For gaze region classification networks, Let be the predicted probability of the j-th gaze region category.

9. The method according to claim 7, characterized in that: Step S6 includes the following sub-steps: Step S601: Based on the geometric transformation relationship obtained when constructing the geometric normalization representation in step S2, establish a preset geometric consistency relationship between the original representation and the prediction result under the geometric normalization representation, and introduce the preset geometric consistency relationship as a constraint term into the joint estimation process; Step S602: After mapping the gaze direction prediction corresponding to the original representation and the gaze direction prediction corresponding to the geometrically normalized representation to the same coordinate system, calculate the consistency error, and construct the geometric consistency loss based on the consistency error. ; Step S603: Construct the line-of-sight regression loss. Compared with gaze region classification loss The line-of-sight regression loss Used to measure and predict line of sight direction Compared to the actual line of sight The difference in the gaze region classification loss Cross-entropy loss is used; Step S604: During the model training or optimization phase, a loss including gaze direction regression is used. gaze area classification loss and geometric consistency loss The joint objective function updates the model parameters, and the joint objective function satisfies: in, The weighting coefficients are used to obtain the final model by minimizing the joint objective function, and the gaze direction estimation result and gaze region classification result optimized by geometric consistency constraints are output.

10. The method according to claim 9, characterized in that: In step S602, the geometric consistency loss is: in, and These are the direction vectors predicted from the line-of-sight directions under the two different representations; In step S603, the gaze direction regression loss satisfy: in, This is a function that maps the line-of-sight direction parameter to a three-dimensional direction vector; The gaze region classification loss satisfy: in, For the number of gaze region categories, For the true label of the gaze area, For the predicted first Class probability.

Images