Depth image normalization method, device and apparatus for line of sight estimation and line of sight estimation system

Abstract

This invention discloses a depth image normalization method, apparatus, device, and gaze estimation system for gaze estimation, relating to the fields of computer vision and 3D data processing. It includes: constructing an original 3D coordinate map based on the original depth map; transforming and inversely mapping the original 3D coordinate map based on a perspective transformation matrix to obtain a normalized 3D coordinate map and the mapped 3D spatial coordinates of each pixel in the normalized 3D coordinate map in the original depth camera coordinate system; transforming the mapped 3D spatial coordinates to the target virtual camera coordinate system through rotation and translation operations in the perspective transformation matrix to obtain the final Z-coordinate of each pixel in the normalized 3D coordinate map, which serves as the normalized depth value for each pixel in the normalized 3D coordinate map; and generating a normalized depth map based on each pixel in the normalized 3D coordinate map and its corresponding normalized depth value, thereby eliminating geometric distortion and achieving accurate normalization of the depth image.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D data processing, and in particular to a depth image normalization method, apparatus, device, and gaze estimation system for gaze estimation. Background Technology

[0002] Appearance-based gaze estimation has become a research hotspot in the field of human-computer interaction in recent years. It aims to directly regress the gaze direction by analyzing face or eye images. To improve the robustness of the model under different head poses, distances, and lighting conditions, it is usually necessary to standardize the input images, transforming them into a uniform and normalized "virtual camera" perspective.

[0003] For gaze estimation tasks, the core idea of ​​current standard image normalization methods is to project all original RGB images, regardless of the camera used or the head pose, onto the image plane of a pre-defined, fixed-parameter virtual camera. Generally, the virtual camera needs to meet the following conditions: the X-axis of the virtual camera is coplanar with the line connecting the human eye; the center point of the face is located on the center line of the virtual camera's field of view; and the distance between the virtual camera and the center point of the face is fixed.

[0004] When incorporating depth information to construct multimodal models, current depth image normalization operations typically treat the depth map as a single-channel grayscale image and employ the same processing methods as RGB images. This involves directly cropping, rotating, and scaling the two-dimensional depth map without considering its core depth values. Consequently, current depth image normalization methods for gaze estimation introduce severe geometric distortion, compromising the physical authenticity of the depth information. Summary of the Invention

[0005] In view of the above-mentioned technical problems, the present invention provides a depth image normalization method, apparatus, device and gaze estimation system for gaze estimation, which aims to overcome the above problems or at least partially solve the above problems.

[0006] The first aspect of this invention provides a depth image normalization method for gaze estimation, the method comprising:

[0007] Based on the original depth map, an original 3D coordinate map is constructed; the pixel value of each pixel in the original 3D coordinate map is the original 3D spatial coordinate of that pixel, which is the 3D spatial coordinate of that pixel in the original depth camera coordinate system;

[0008] Based on the perspective transformation matrix, the original 3D coordinate map is transformed and reverse mapped to obtain a standardized 3D coordinate map, and the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system are obtained.

[0009] The mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system are transformed to the target virtual camera coordinate system through the rotation matrix and translation operation in the perspective transformation matrix to obtain the final Z coordinate of each pixel in the standardized 3D coordinate map, which is used as the standardized depth value of each pixel in the standardized 3D coordinate map.

[0010] A standardized depth map is generated based on each pixel and its corresponding standardized depth value in the standardized 3D coordinate map.

[0011] A second aspect of the present invention provides a gaze estimation system, wherein the multimodal input of the gaze estimation system includes at least: a standardized depth map and a standardized RGB image;

[0012] The standardized depth map is obtained by a depth image standardization method for line-of-sight estimation as described in the first aspect of the present invention.

[0013] A third aspect of the present invention provides a depth image normalization apparatus for gaze estimation, the apparatus comprising:

[0014] An image construction module is used to construct an original 3D coordinate map based on the original depth map; the pixel value of each pixel in the original 3D coordinate map is the original 3D spatial coordinate of the pixel, which is the 3D spatial coordinate of the pixel in the original depth camera coordinate system;

[0015] The reverse mapping module is used to transform and reverse map the original 3D coordinate map based on the perspective transformation matrix to obtain a standardized 3D coordinate map, and the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system.

[0016] The depth value normalization module is used to transform the mapped three-dimensional space coordinates of each pixel in the normalized three-dimensional coordinate map under the original depth camera coordinate system to the target virtual camera coordinate system through the rotation matrix and translation operation in the perspective transformation matrix, so as to obtain the final Z coordinate of each pixel in the normalized three-dimensional coordinate map, which is used as the normalized depth value of each pixel in the normalized three-dimensional coordinate map.

[0017] The depth map generation module is used to generate a standardized depth map based on each pixel and its corresponding standardized depth value in the standardized 3D coordinate map.

[0018] A fourth aspect of the present invention provides an electronic device comprising a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the depth image normalization method for gaze estimation as described in the first aspect of the present invention.

[0019] In the depth image normalization method for gaze estimation proposed in this invention, the original depth map is converted into an original 3D coordinate map with pixel values ​​as original 3D spatial coordinates. A perspective transformation is then performed on the original 3D coordinate map to normalize the pixel coordinates in the 2D plane, resulting in a normalized 3D coordinate map. Through reverse mapping, the mapped 3D spatial coordinates of each pixel in the normalized 3D coordinate map under the original depth camera coordinate system are obtained, serving as the final pixel value for each pixel in the normalized 3D coordinate map. Each final pixel value in the normalized 3D coordinate map is then transformed to the target virtual camera coordinate system through a rotation matrix and translation operation, and its Z-coordinate is taken as the final depth value, achieving depth value normalization. Finally, a normalized depth map is generated based on each pixel in the normalized 3D coordinate map and its corresponding normalized depth value. Thus, this invention, through the aforementioned depth map normalization process based on reverse mapping of the "3D coordinate map," ensures that each normalized depth value is obtained after precise 3D geometric transformation, fundamentally eliminating geometric distortion caused by related depth image normalization methods. This achieves precise depth image normalization, providing physically and geometrically accurate 3D face surface information for subsequent gaze estimation, achieving high-quality input, and further improving the accuracy of gaze estimation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the steps of a depth image normalization method for line-of-sight estimation according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart illustrating a depth image normalization method based on a three-dimensional coordinate graph, according to an embodiment of the present invention.

[0023] Figure 3 This is a structural block diagram of a depth image normalization device for line-of-sight estimation provided in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Current depth image normalization methods simply treat depth maps as 2D grayscale images and perform perspective transformations, which is geometrically incorrect. This is because each pixel value d in the depth map, along with its pixel coordinates (u, v), defines a 3D point in space. When the viewpoint (i.e., the virtual camera) changes, the depth value (i.e., the Z-coordinate) of this 3D point under the new viewpoint will inevitably change. However, these methods do not update the depth value accordingly, resulting in severe distortion and deformation of the 3D face model represented by the normalized depth map. This geometric distortion is fatal for gaze estimation, a task highly dependent on the accurate 3D geometry of the eye region (such as orbital depth and corneal curvature). It inputs incorrect geometric priors into deep learning models (such as gaze estimation models or systems), severely limiting the performance ceiling of multimodal fusion and preventing the model from learning the correct "shape-gaze" association.

[0027] In addition, some intuitive improvement methods, such as forward projection (projecting the original point cloud onto a new plane), can produce large data holes in practical applications due to resolution mismatch. These holes are difficult to fill and are prone to artifacts, and thus cannot provide high-quality input.

[0028] Based on this, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, this invention proposes the first solution in the field of gaze estimation to solve the depth map standardization problem through three-dimensional coordinate map transformation. Specifically, embodiments of this invention propose a depth image standardization method for gaze estimation that can efficiently generate dense standardized depth maps while strictly maintaining three-dimensional geometric consistency. In this method, through a depth map standardization process based on the reverse mapping of "three-dimensional coordinate map", it can be ensured that each standardized depth value is obtained after accurate three-dimensional geometric transformation, fundamentally eliminating the geometric distortion caused by related depth image standardization methods, ensuring geometric consistency, solving the problem of serious geometric distortion and data gaps introduced by current depth image standardization methods in gaze estimation tasks, realizing accurate standardization of depth images, providing physically and geometrically accurate three-dimensional face surface information for subsequent gaze estimation, achieving high-quality input, and further improving the accuracy of gaze estimation.

[0029] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a depth image normalization method for gaze estimation according to an embodiment of the present invention. Figure 1 As shown, the depth image normalization method for gaze estimation provided in this embodiment includes at least the following steps:

[0030] Step S11: Construct the original 3D coordinate map based on the original depth map.

[0031] In this embodiment, an original 3D coordinate map corresponding to the original depth map can be constructed based on the original depth map. The original depth map is a depth map to be normalized, where each pixel's pixel value is its corresponding depth value. The original 3D coordinate map shows the original 3D spatial coordinates of each pixel in the original depth camera coordinate system. In this embodiment, the original depth camera coordinate system is a 3D coordinate system with the optical center of the depth camera acquiring the original depth map as the origin and the optical axis of the depth camera as the Z-axis.

[0032] In this embodiment, the pixel coordinates of each pixel in the original depth map are the same as the pixel coordinates of each pixel in the original 3D coordinate map. The pixel coordinates in this embodiment are represented by (u, v) coordinates.

[0033] Step S12: Transform and reverse map the original 3D coordinate map based on the perspective transformation matrix to obtain a standardized 3D coordinate map, and the mapped 3D spatial coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system.

[0034] In this embodiment, a perspective transformation matrix is ​​used to transform the original 3D coordinate map to obtain a standardized 3D coordinate map. This perspective transformation matrix is ​​used to achieve depth image standardization, aiming to transform the original 3D coordinate map to the target virtual camera coordinate system; the standardized 3D coordinate map is a 3D coordinate map in the target virtual camera coordinate system.

[0035] The perspective transformation matrix only transforms the two-dimensional space (i.e., pixel coordinates) in the original 3D coordinate map, without changing the pixel values ​​(i.e., original 3D spatial coordinates) in the original 3D coordinate map. Therefore, this embodiment performs a reverse mapping of the standardized 3D coordinate map based on the perspective transformation matrix to obtain the mapped 3D spatial coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system. These mapped 3D spatial coordinates are the original 3D spatial coordinates of pixels whose pixel coordinates have been changed in the standardized 3D coordinate map, as defined in the original depth camera coordinate system.

[0036] In a specific example, the pixel coordinates of pixel A in the original 3D coordinate map are (2, 3), and the pixel value corresponding to pixel A is (1, 2, 3). The pixel coordinates of pixel A in the standardized 3D coordinate map obtained by transforming the original 3D coordinate map based on the perspective transformation matrix become (3, 1). However, the intermediate pixel value corresponding to pixel A in the standardized 3D coordinate map is (3, 5, 6), which is the pixel value corresponding to pixel coordinate (3, 1) in the original 3D coordinate map. Based on this, the standardized 3D coordinate map can be reverse-mapped through the perspective transformation matrix to obtain the final pixel value of pixel A (pixel coordinate (3, 1)) in the standardized 3D coordinate map. That is, the mapped 3D space coordinates of pixel A (pixel coordinate (3, 1)) in the original depth camera coordinate system are (1, 2, 3).

[0037] In an optional embodiment, the perspective transformation matrix can be calculated based on the head pose (such as rotation vector hr and / or translation vector ht) in the original depth map, the depth camera intrinsics corresponding to the original depth map (including the depth camera intrinsic matrix corresponding to the original depth map), and the preset target virtual camera intrinsics. In this embodiment, the process of determining the perspective transformation matrix follows standard methods in the field of gaze estimation: aiming to transform the face image to a preset virtual camera viewpoint with a fixed distance and pose.

[0038] Step S13: Transform the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map to the target virtual camera coordinate system through the rotation matrix and translation operation in the perspective transformation matrix, so as to obtain the final Z coordinate of each pixel in the standardized 3D coordinate map, which is used as the standardized depth value of each pixel in the standardized 3D coordinate map.

[0039] In this embodiment, each pixel in the standardized 3D coordinate map can be traversed to obtain the final pixel value of each pixel, i.e., the mapped 3D space coordinates corresponding to each pixel in the original depth camera coordinate system. Then, using the rotation matrix in the perspective transformation matrix and a translation operation, the mapped 3D space coordinates corresponding to each pixel in the original depth camera coordinate system are transformed to the target virtual camera coordinate system, obtaining the final 3D space coordinates corresponding to each pixel in the standardized 3D coordinate map. The Z coordinate in the final 3D space coordinates is determined as the final Z coordinate, thus obtaining the final Z coordinate corresponding to each pixel in the standardized 3D coordinate map. The final Z coordinate corresponding to each pixel in the standardized 3D coordinate map is used as the standardized depth value corresponding to each pixel in the standardized 3D coordinate map, thereby achieving the standardization of the depth value corresponding to the pixel. In this embodiment, the standardized depth value is the geometrically completely correct, standardized depth value.

[0040] Step S14: Generate a standardized depth map based on each pixel and its corresponding standardized depth value in the standardized 3D coordinate map.

[0041] In this embodiment, after obtaining the pixel coordinates of each pixel in the standardized 3D coordinate map and the standardized depth value corresponding to each pixel, a single-channel standardized depth map can be generated based on the pixel coordinates of each pixel in the standardized 3D coordinate map and its corresponding standardized depth value, thus completing the standardization of the original depth map.

[0042] In this embodiment, the original depth map is converted into an original 3D coordinate map with pixel values ​​as original 3D spatial coordinates. A perspective transformation is then performed on the original 3D coordinate map to standardize the pixel coordinates in the 2D plane, resulting in a standardized 3D coordinate map. A reverse mapping is used to obtain the mapped 3D spatial coordinates of each pixel in the standardized 3D coordinate map within the original depth camera coordinate system, serving as the final pixel value for each pixel in the standardized 3D coordinate map. Each final pixel value in the standardized 3D coordinate map is then transformed to the target virtual camera coordinate system through a rotation matrix and translation operation, and its Z-coordinate is taken as the final depth value, achieving depth value standardization. Finally, a standardized depth map is generated based on each pixel in the standardized 3D coordinate map and its corresponding standardized depth value. Thus, this invention, through the aforementioned depth map standardization process based on a reverse mapping of a "3D coordinate map," ensures that each standardized depth value is obtained after precise 3D geometric transformation, fundamentally eliminating geometric distortion caused by related depth image standardization methods. This achieves precise standardization of the depth image, providing physically and geometrically accurate 3D face surface information for subsequent gaze estimation, achieving high-quality input, and further improving the accuracy of gaze estimation. In other words, this method is a depth image processing approach for the field of gaze estimation that maintains the consistency of 3D geometry during the standardization process before training.

[0043] In conjunction with the above embodiments, in one implementation, the present invention also provides a depth image normalization method for gaze estimation, in which step S11 may specifically include steps S21 and S22:

[0044] Step S21: Traverse each pixel in the original depth map, and use the depth value and pixel coordinates of the pixel, as well as the depth camera intrinsic parameter matrix corresponding to the original depth map, to determine the original three-dimensional spatial coordinates of the pixel.

[0045] In this embodiment, for the original depth map, each pixel in the original depth map is traversed. Using the depth value and pixel coordinates (u, v) of that pixel, as well as the depth camera intrinsic parameter matrix corresponding to the original depth map, the original 3D spatial coordinates (x, y, z) of that pixel are calculated. The depth camera intrinsic parameter matrix corresponding to the original depth map is the camera intrinsic parameter matrix of the depth camera that acquired the original depth map. The z-coordinate in the original 3D spatial coordinates is the depth value of that pixel.

[0046] Step S22: Use the original three-dimensional spatial coordinates corresponding to each pixel in the original depth map as the pixel value of that pixel to construct the original three-dimensional coordinate map.

[0047] In this embodiment, the original three-dimensional spatial coordinates (x, y, z) corresponding to each pixel in the original depth map can be used as the pixel value of the original three-dimensional coordinate map at the pixel position (u, v). That is, the original three-dimensional spatial coordinates corresponding to each pixel in the original depth map are used as the pixel value of that pixel. Then, based on the pixel coordinates and pixel values ​​of each pixel in the original depth map, the original three-dimensional coordinate map is constructed.

[0048] In this embodiment, the "original 3D coordinate map" is a new floating-point array with the same size as the original depth map but having three channels. It can be understood that the original 3D coordinate map is a three-channel image with the same size as the original depth map.

[0049] In conjunction with any of the above embodiments, the present invention also provides a depth image normalization method for gaze estimation, wherein step S12 may specifically include steps S31 to S32:

[0050] Step S31: Based on the perspective transformation matrix, the original three-dimensional coordinate map is transformed by the perspective transformation function to obtain the standardized three-dimensional coordinate map.

[0051] In this embodiment, the perspective transformation matrix itself is used to standardize the RGB image (a three-channel image with RGB values ​​as pixel values). Based on this, this embodiment treats the original three-dimensional coordinate map as a three-channel image, and uses the perspective transformation matrix to transform the original three-dimensional coordinate map through a perspective transformation function to obtain a standardized three-dimensional coordinate map.

[0052] In one optional example, the perspective transformation function can be a standard image perspective transformation function, such as cv2.warpPerspective, without any specific restrictions.

[0053] Step S32: For each pixel in the standardized 3D coordinate map, the corresponding mapped 3D spatial coordinates of the pixel in the original depth camera coordinate system are obtained by reverse mapping using the perspective transformation function and the perspective transformation matrix.

[0054] In this embodiment, after obtaining the standardized 3D coordinate map, for each pixel (u', v') in the standardized 3D coordinate map, the corresponding mapped 3D spatial coordinates (X, Y, Z) in the original depth camera coordinate system can be calculated by reverse mapping using the perspective transformation function and perspective transformation matrix in step S31. The pixel coordinates in the standardized 3D coordinate map are represented by (u', v'). The mapped 3D spatial coordinates corresponding to each pixel in the standardized 3D coordinate map in the original depth camera coordinate system represent the 3D spatial coordinates corresponding to the (u', v') position of that pixel in the standardized 3D coordinate map in the original depth camera coordinate system.

[0055] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a depth image normalization method for gaze estimation. In this method, step S32 specifically includes steps S41 to S42:

[0056] Step S41: For each pixel in the standardized 3D coordinate map, the pixel coordinates corresponding to that pixel in the original 3D coordinate map are obtained by reverse mapping using the perspective transformation function and the perspective transformation matrix.

[0057] In this embodiment, for each pixel (u', v') in the standardized three-dimensional coordinate map, the non-integer coordinates corresponding to the pixel in the original three-dimensional coordinate map are first calculated by reverse mapping using the perspective transformation function and perspective transformation matrix in step S31. These non-integer coordinates are the pixel coordinates in the original three-dimensional coordinate map.

[0058] Step S42: Based on the pixel coordinates of each pixel in the standardized 3D coordinate map corresponding to the pixel coordinates in the original 3D coordinate map, determine the neighboring pixels of the pixel; then, based on the original 3D spatial coordinates of the neighboring pixels of the pixel corresponding to the original 3D coordinate map, perform bilinear interpolation to obtain the mapped 3D spatial coordinates of the pixel in the original depth camera coordinate system.

[0059] In this embodiment, based on the pixel coordinates (u0, v0) of each pixel in the standardized three-dimensional coordinate map corresponding to the pixel coordinates in the original three-dimensional coordinate map (e.g., (u0, v0), which are non-integer coordinates), the integer pixel coordinates around the pixel coordinates (u0, v0) in the original three-dimensional coordinate map can be determined, and the pixels corresponding to the integer pixel coordinates can be determined as the neighboring pixels of the pixel (u0, v0).

[0060] Then, based on the original 3D spatial coordinates of the neighboring pixels of the pixel in the original 3D coordinate map, bilinear interpolation is performed to obtain an accurate interpolated 3D spatial coordinate, which is used as the mapped 3D spatial coordinate of the pixel in the original depth camera coordinate system. This process yields the mapped 3D spatial coordinates of each pixel in the standardized 3D coordinate map in the original depth camera coordinate system.

[0061] Compared to the forward projection method, this embodiment utilizes the reverse mapping mechanism of the graphics library optimization function (i.e., perspective transformation function) to generate a fully dense standardized depth map without any post-processing filling, thus solving the data hole problem caused by the current depth map standardization method.

[0062] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a depth image normalization method for gaze estimation. In this method, step S13 above, "transforming the mapped 3D spatial coordinates of each pixel in the normalized 3D coordinate map under the original depth camera coordinate system into the target virtual camera coordinate system through rotation and translation operations in the perspective transformation matrix, to obtain the final Z-coordinate corresponding to each pixel in the normalized 3D coordinate map," specifically includes steps S51 to S52:

[0063] Step S51: Transform the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system using the rotation matrix in the perspective transformation matrix to obtain the intermediate 3D space coordinates of that pixel.

[0064] In this embodiment, the rotation matrix in the perspective transformation matrix is ​​the rotation matrix obtained when calculating the perspective transformation matrix. First, the mapped 3D spatial coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system can be transformed using the rotation matrix in the perspective transformation matrix to complete the rotation part of the standardization process. This step allows the target virtual camera to directly face the center point of the head, obtaining the intermediate 3D spatial coordinates (x', y', z'') corresponding to that pixel.

[0065] Step S52: Based on the translation distance, perform a translation operation on the Z coordinate in the intermediate three-dimensional space coordinates corresponding to each pixel in the standardized three-dimensional coordinate map to obtain the final Z coordinate corresponding to each pixel in the standardized three-dimensional coordinate map.

[0066] In this embodiment, since the perspective transformation matrix is ​​standardized for RGB images, and RGB images are two-dimensional planes without depth variations, the perspective transformation matrix only includes rotation and scaling matrices, and does not include translation. Therefore, this embodiment performs a translation operation on the Z-coordinate (z'') of each pixel in the standardized 3D coordinate map based on the translation distance. This step causes the target virtual camera to translate along the line connecting the camera and the head center point to a pre-set distance (i.e., translation distance), obtaining the final Z-coordinate (z') of each pixel in the standardized 3D coordinate map, and thus obtaining the transformed 3D spatial coordinates (x', y', z') of each pixel in the standardized 3D coordinate map in the target virtual camera coordinate system. The final Z-coordinate (z') of each pixel in the standardized 3D coordinate map is the standardized depth value of each pixel in the standardized 3D coordinate map.

[0067] In one optional embodiment, the final Z-coordinate of each pixel in the standardized 3D coordinate map can be obtained by subtracting the translation distance d from the Z-coordinate in the intermediate 3D space coordinates corresponding to each pixel. That is, z' = z'' - d.

[0068] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a depth image normalization method for gaze estimation. In this method, in addition to the steps described above, steps S61 to S62 may also be included:

[0069] Step S61: Based on the original three-dimensional spatial coordinates corresponding to the head center point in the original depth map, obtain the first distance between the original camera and the head center point.

[0070] In this embodiment, the first distance between the original camera and the head center point can be calculated based on the original three-dimensional spatial coordinates corresponding to the pixels corresponding to the head center point in the original depth map. The original camera is the depth camera that acquired the original depth map.

[0071] Step S62: Based on the first distance and the distance between the target virtual camera and the center point of the head in the target virtual camera intrinsic parameters, obtain the translation distance.

[0072] In this embodiment, the preset intrinsic parameters of the target virtual camera include the distance between the target virtual camera and the center point of the head. The translation distance can be obtained based on the first distance and the distance between the target virtual camera and the center point of the head. In an optional example, the translation distance is the difference between the first distance and the distance between the target virtual camera and the center point of the head.

[0073] In one embodiment, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating a depth image normalization method based on a three-dimensional coordinate map, as shown in an embodiment of the present invention. Figure 2 In China, depth image normalization methods based on 3D coordinate maps include:

[0074] Step 101: Construct the original 3D coordinate map. In this step, an original 3D coordinate map (i.e., a 3D depth map) with the same size as the original depth map is constructed based on the original depth map. The pixel values ​​in the original 3D coordinate map are the original 3D spatial coordinates, and the Z coordinate in the original 3D spatial coordinates represents the depth value. Figure 2 The left part is the original 3D coordinate map that was constructed. The grayscale image containing the face in the left part is a depth map formed by the Z coordinate in the original 3D spatial coordinates.

[0075] Step 102: Obtain the mapped 3D space coordinates of pixels in the standardized 3D coordinate map in the original 3D coordinate map using the perspective transformation matrix. This step involves transforming and reversing the original 3D coordinate map using the perspective transformation matrix to obtain the standardized 3D coordinate map and the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map in the original 3D coordinate map.

[0076] Step 103: Obtain the standardized spatial position of each pixel in the standardized 3D coordinate map through rotation matrix and translation operations. This step transforms the mapped 3D spatial coordinates of each pixel in the standardized 3D coordinate map to the target virtual camera coordinate system through rotation matrix and translation operation in the perspective transformation matrix, obtaining the transformed 3D spatial coordinates (x', y', z'), i.e., the standardized spatial position. This yields the final Z-coordinate of each pixel in the standardized 3D coordinate map, which serves as the standardized depth value for each pixel in the standardized 3D coordinate map. Finally, the standardized depth values ​​of all pixels in the standardized 3D coordinate map are combined to form the final, dense, standardized single-channel depth map (e.g., ...). Figure 2 The right side of the image shows a standardized depth image, which is used to standardize the depth map for subsequent gaze estimation.

[0077] It should be noted that, Figure 2 In this context, (u', v') represents the standardized pixel coordinates of a feature point (e.g., the feature point corresponding to the right eyelid), and (x', y', z') represents the standardized three-dimensional spatial coordinates of the same feature point (e.g., the feature point corresponding to the right eyelid). (u, v) represents the pixel coordinates of the same feature point (e.g., the feature point corresponding to the right eyelid) before standardization, and (x, y, z) represents the three-dimensional spatial coordinates of the same feature point (e.g., the feature point corresponding to the right eyelid) before standardization.

[0078] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a depth image normalization method for gaze estimation. In addition to the steps described above, this method may further include step S71:

[0079] Step S71: Input the standardized depth map and the standardized RGB image as multimodal inputs into the gaze estimation system to perform gaze estimation.

[0080] In this embodiment, after obtaining the standardized depth map, the standardized depth map and the standardized RGB image can be used as multimodal inputs to the gaze estimation system for gaze estimation. This gaze estimation system can be a pre-trained gaze estimation system or a gaze estimation system to be trained; this embodiment does not impose any restrictions. The standardized RGB image is obtained by standardizing the original RGB image corresponding to the original depth map, and the standardized depth map and the standardized RGB image are located in the same target virtual camera coordinate system.

[0081] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0082] Based on the same inventive concept, one embodiment of the present invention provides a gaze estimation system. The multimodal input of the gaze estimation system includes at least: a standardized depth map and a standardized RGB image; wherein, the standardized depth map is obtained by the depth image standardization method for gaze estimation described in any of the above embodiments of the present invention.

[0083] Based on the same inventive concept, one embodiment of the present invention provides a depth image normalization apparatus for gaze estimation. (Reference) Figure 3 , Figure 3 This is a structural block diagram of a depth image normalization device for gaze estimation provided in an embodiment of the present invention. Figure 3 As shown, the device includes:

[0084] An image construction module is used to construct an original 3D coordinate map based on the original depth map; the pixel value of each pixel in the original 3D coordinate map is the original 3D spatial coordinate of the pixel, which is the 3D spatial coordinate of the pixel in the original depth camera coordinate system;

[0085] The reverse mapping module is used to transform and reverse map the original 3D coordinate map based on the perspective transformation matrix to obtain a standardized 3D coordinate map, and the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system.

[0086] The depth value normalization module is used to transform the mapped three-dimensional space coordinates of each pixel in the normalized three-dimensional coordinate map under the original depth camera coordinate system to the target virtual camera coordinate system through the rotation matrix and translation operation in the perspective transformation matrix, so as to obtain the final Z coordinate of each pixel in the normalized three-dimensional coordinate map, which is used as the normalized depth value of each pixel in the normalized three-dimensional coordinate map.

[0087] The depth map generation module is used to generate a standardized depth map based on each pixel and its corresponding standardized depth value in the standardized 3D coordinate map.

[0088] Optionally, the image building module includes:

[0089] The first determining module is used to traverse each pixel in the original depth map, and use the depth value and pixel coordinates of the pixel, as well as the depth camera intrinsic parameter matrix corresponding to the original depth map, to determine the original three-dimensional spatial coordinates of the pixel.

[0090] The first construction module is used to construct the original three-dimensional coordinate map by taking the original three-dimensional spatial coordinates corresponding to each pixel in the original depth map as the pixel value of that pixel. The original three-dimensional coordinate map is a three-channel image with the same size as the original depth map.

[0091] Optionally, the reverse mapping module includes:

[0092] The transformation module is used to transform the original three-dimensional coordinate map based on the perspective transformation matrix and through the perspective transformation function to obtain the standardized three-dimensional coordinate map;

[0093] The mapping module is used to reverse map each pixel in the standardized 3D coordinate map to obtain the mapped 3D space coordinates of the pixel in the original depth camera coordinate system by using the perspective transformation function and the perspective transformation matrix.

[0094] In the standardized 3D coordinate map, the mapped 3D spatial coordinates of each pixel in the original depth camera coordinate system represent the 3D spatial coordinates corresponding to the position of that pixel in the standardized 3D coordinate map in the original depth camera coordinate system.

[0095] Optionally, the mapping module includes:

[0096] The second determining module is used to obtain the intermediate three-dimensional space coordinates of each pixel in the original three-dimensional coordinate map by reverse mapping using the perspective transformation function and the perspective transformation matrix for each pixel in the standardized three-dimensional coordinate map.

[0097] The third determining module is used to perform bilinear interpolation based on the intermediate three-dimensional space coordinates of each pixel in the standardized three-dimensional coordinate map corresponding to the original three-dimensional coordinate map, and the intermediate three-dimensional space coordinates of the neighboring pixels of the pixel corresponding to the original three-dimensional coordinate map, to obtain the mapped three-dimensional space coordinates of the pixel in the original depth camera coordinate system.

[0098] Optionally, the depth value normalization module includes:

[0099] The rotation module is used to transform the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system through the rotation matrix in the perspective transformation matrix to obtain the intermediate 3D space coordinates of the pixel.

[0100] The translation module is used to perform a translation operation on the Z coordinate of each pixel in the intermediate three-dimensional space coordinates of the standardized three-dimensional coordinate map based on the translation distance, so as to obtain the final Z coordinate of each pixel in the standardized three-dimensional coordinate map.

[0101] Optionally, the device further includes:

[0102] The first distance determination module is used to obtain the first distance between the original camera and the head center point based on the original three-dimensional spatial coordinates corresponding to the head center point in the original depth map.

[0103] The translation distance determination module is used to obtain the translation distance based on the first distance and the distance between the target virtual camera and the center point of the head in the target virtual camera intrinsic parameters.

[0104] Optionally, the device further includes:

[0105] The gaze estimation module is used to input the standardized depth map and the standardized RGB image as multimodal inputs to the gaze estimation system for gaze estimation.

[0106] The standardized RGB image is obtained by standardizing the original RGB image corresponding to the original depth map, and the standardized depth map and the standardized RGB image are in the same target virtual camera coordinate system.

[0107] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0108] The depth image normalization device for gaze estimation in this embodiment of the invention can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment of the invention does not impose specific limitations.

[0109] The depth image normalization device for gaze estimation in this embodiment of the invention can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment of the invention does not impose specific limitations.

[0110] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps in the depth image normalization method for gaze estimation described in any of the above embodiments of the present invention.

[0111] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0112] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0113] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0115] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A depth image normalization method for gaze estimation, characterized in that, The method includes: Based on the original depth map, an original 3D coordinate map is constructed; the pixel value of each pixel in the original 3D coordinate map is the original 3D spatial coordinate of that pixel, which is the 3D spatial coordinate of that pixel in the original depth camera coordinate system; Based on the perspective transformation matrix, the original 3D coordinate map is transformed and reverse mapped to obtain a standardized 3D coordinate map, and the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system are obtained. The mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system are transformed to the target virtual camera coordinate system through the rotation matrix and translation operation in the perspective transformation matrix to obtain the final Z coordinate of each pixel in the standardized 3D coordinate map, which is used as the standardized depth value of each pixel in the standardized 3D coordinate map. Based on each pixel and its corresponding standardized depth value in the standardized 3D coordinate map, a standardized depth map is generated; The mapped 3D spatial coordinates of each pixel in the standardized 3D coordinate map, under the original depth camera coordinate system, are transformed to the target virtual camera coordinate system through rotation and translation operations in the perspective transformation matrix, resulting in the final Z-coordinate of each pixel in the standardized 3D coordinate map, including: The mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system are transformed by the rotation matrix in the perspective transformation matrix to obtain the intermediate 3D space coordinates of the pixel. Based on the original three-dimensional spatial coordinates corresponding to the head center point in the original depth map, the first distance between the original camera and the head center point is obtained. Based on the first distance and the distance between the target virtual camera and the center point of the head in the target virtual camera intrinsic parameters, the translation distance is obtained; Based on the translation distance, the Z coordinate in the intermediate three-dimensional space coordinates corresponding to each pixel in the standardized three-dimensional coordinate map is translated to obtain the final Z coordinate corresponding to each pixel in the standardized three-dimensional coordinate map.

2. The depth image normalization method for gaze estimation according to claim 1, characterized in that, Based on the original depth map, construct the original 3D coordinate map, including: By traversing each pixel in the original depth map, the original three-dimensional spatial coordinates corresponding to the pixel are determined using the depth value and pixel coordinates of the pixel, as well as the depth camera intrinsic parameter matrix corresponding to the original depth map. The original three-dimensional spatial coordinates corresponding to each pixel in the original depth map are used as the pixel value of that pixel to construct the original three-dimensional coordinate map, which is a three-channel image with the same size as the original depth map.

3. The depth image normalization method for gaze estimation according to claim 1, characterized in that, The original 3D coordinate map is transformed and reverse-mapped based on the perspective transformation matrix to obtain a standardized 3D coordinate map. The standardized 3D coordinate map contains the mapped 3D spatial coordinates of each pixel in the original depth camera coordinate system, including: Based on the perspective transformation matrix, the original three-dimensional coordinate map is transformed by the perspective transformation function to obtain the standardized three-dimensional coordinate map; For each pixel in the standardized 3D coordinate map, the corresponding mapped 3D spatial coordinates of the pixel in the original depth camera coordinate system are obtained by reverse mapping using the perspective transformation function and the perspective transformation matrix. In the standardized 3D coordinate map, the mapped 3D spatial coordinates of each pixel in the original depth camera coordinate system represent the 3D spatial coordinates corresponding to the position of that pixel in the standardized 3D coordinate map in the original depth camera coordinate system.

4. The depth image normalization method for gaze estimation according to claim 3, characterized in that, For each pixel in the standardized 3D coordinate map, the mapped 3D spatial coordinates of that pixel in the original depth camera coordinate system are obtained by reverse mapping using the perspective transformation function and the perspective transformation matrix, including: For each pixel in the standardized 3D coordinate graph, the pixel coordinates corresponding to that pixel in the original 3D coordinate graph are obtained by reverse mapping using the perspective transformation function and the perspective transformation matrix; Based on the pixel coordinates of each pixel in the standardized 3D coordinate map corresponding to the pixel coordinates in the original 3D coordinate map, the neighboring pixels of the pixel are determined; then, based on the original 3D spatial coordinates of the neighboring pixels of the pixel corresponding to the original 3D coordinate map, bilinear interpolation is performed to obtain the mapped 3D spatial coordinates of the pixel in the original depth camera coordinate system.

5. The depth image normalization method for gaze estimation according to any one of claims 1 to 4, characterized in that, The method further includes: The standardized depth map and the standardized RGB image are used as multimodal inputs and fed into the gaze estimation system for gaze estimation. The standardized RGB image is obtained by standardizing the original RGB image corresponding to the original depth map, and the standardized depth map and the standardized RGB image are in the same target virtual camera coordinate system.

6. A gaze estimation system, characterized in that, The multimodal input of the gaze estimation system includes at least: a standardized depth map and a standardized RGB image; The standardized depth map is obtained by the depth image standardization method for line-of-sight estimation as described in any one of claims 1 to 5.

7. A depth image normalization device for gaze estimation, characterized in that, The device includes: An image construction module is used to construct an original 3D coordinate map based on the original depth map; the pixel value of each pixel in the original 3D coordinate map is the original 3D spatial coordinate of the pixel, which is the 3D spatial coordinate of the pixel in the original depth camera coordinate system; The reverse mapping module is used to transform and reverse map the original 3D coordinate map based on the perspective transformation matrix to obtain a standardized 3D coordinate map, and the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system. The depth value normalization module is used to transform the mapped three-dimensional space coordinates of each pixel in the normalized three-dimensional coordinate map under the original depth camera coordinate system to the target virtual camera coordinate system through the rotation matrix and translation operation in the perspective transformation matrix, so as to obtain the final Z coordinate of each pixel in the normalized three-dimensional coordinate map, which is used as the normalized depth value of each pixel in the normalized three-dimensional coordinate map. The depth map generation module is used to generate a standardized depth map based on each pixel and its corresponding standardized depth value in the standardized 3D coordinate map. The depth value standardization module is further configured to transform the mapped 3D space coordinates of each pixel in the standardized 3D coordinate map under the original depth camera coordinate system using the rotation matrix in the perspective transformation matrix to obtain the intermediate 3D space coordinates corresponding to that pixel; based on the original 3D space coordinates corresponding to the head center point in the original depth map, obtain the first distance between the original camera and the head center point; based on the first distance and the distance between the target virtual camera and the head center point in the target virtual camera intrinsic parameters, obtain the translation distance; based on the translation distance, perform a translation operation on the Z coordinate in the intermediate 3D space coordinates corresponding to each pixel in the standardized 3D coordinate map to obtain the final Z coordinate corresponding to each pixel in the standardized 3D coordinate map.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the depth image normalization method for gaze estimation as described in any one of claims 1 to 5.

Images