Method and device for determining three-dimensional key points, electronic equipment and medium

Abstract

The invention provides a method and device for determining three-dimensional key points, electronic equipment and a medium. According to the specific implementation scheme, the method comprises the steps of obtaining a binocular image which is collected by a real binocular camera and contains a target object; determining a two-dimensional coordinate of the target object in the binocular image; based on a conversion matrix from a camera coordinate system of the real binocular camera to a camera coordinate system of the virtual binocular camera, converting the two-dimensional coordinates of the target object to corresponding normalized planes of the virtual binocular camera, and obtaining normalized plane coordinates of the target object; wherein the virtual binocular camera is a parallel binocular camera; according to the normalized plane coordinate of the target object and a three-dimensional key point prediction model of the virtual standard binocular camera, obtaining a three-dimensional coordinate of the target object in a camera coordinate system of the virtual binocular camera; and determining the three-dimensional coordinate of the target object in the camera coordinate system of the real binocular camera according to the three-dimensional coordinate of the target object in the camera coordinate system of the virtual binocular camera.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device, electronic device, and medium for determining three-dimensional key points. Background Art

[0002] Three-dimensional (3D) pose estimation of objects is widely used in scenarios such as virtual reality (VR), human-computer interaction, motion recognition, and posture recognition of workers in hazardous locations. 3D pose estimation typically requires determining the 3D key points of an object. Summary of the Invention

[0003] In a first aspect, an embodiment of the present disclosure provides a method for determining three-dimensional key points, comprising: obtaining a binocular image containing a target object captured by a real binocular camera; determining the two-dimensional coordinates of the target object in the binocular image; converting the two-dimensional coordinates of the target object in the binocular image to the normalized planes of the corresponding virtual binocular camera based on a transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of a virtual binocular camera, thereby obtaining the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera; wherein the virtual binocular camera is a parallel binocular camera; obtaining the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and a three-dimensional key point prediction model of a virtual standard binocular camera; wherein the virtual standard binocular camera is a parallel binocular camera with a preset baseline length; and determining the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera based on the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0004] In a second aspect, an embodiment of the present disclosure provides a device for determining three-dimensional key points, comprising: an image acquisition module for acquiring a binocular image containing a target object captured by a real binocular camera; a two-dimensional coordinate determination module for determining the two-dimensional coordinates of the target object in the binocular image; a normalization module for converting the two-dimensional coordinates of the target object in the binocular image to the corresponding normalized planes of the virtual binocular camera based on a conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, thereby obtaining the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera; In the embodiment, the virtual binocular camera is a parallel binocular camera; the virtual three-dimensional coordinate determination module is used to obtain the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera according to the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and the three-dimensional key point prediction model of the virtual standard binocular camera; wherein the virtual standard binocular camera is a parallel binocular camera with a preset baseline length; the real three-dimensional coordinate determination module is used to determine the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera according to the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0005] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program for executing the method for determining three-dimensional key points provided by any of the above embodiments of the present disclosure.

[0006] In a fourth aspect, an embodiment of the present disclosure provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the method for determining three-dimensional key points provided in any of the above embodiments of the present disclosure.

[0007] In a fifth aspect, an embodiment of the present disclosure provides a head-mounted display device, the head-mounted display device comprising:

[0008] A processor; a memory for storing processor-executable instructions; a processor for reading the executable instructions from the memory and executing the instructions to implement the method for determining three-dimensional key points provided in any of the above embodiments of the present disclosure.

[0009] In a sixth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer program instructions. When the computer program instructions are executed by a processor, the processor executes the method for determining three-dimensional key points provided in any of the above embodiments.

[0010] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features, and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 is an exemplary system architecture that can be applied to some embodiments of the method or apparatus for determining three-dimensional key points of the present disclosure;

[0013] Figure 2 is a flowchart of a method for determining three-dimensional key points provided by some exemplary embodiments of the present disclosure;

[0014] Figure 3 is a schematic diagram of a process for determining a conversion matrix from a camera coordinate system of a real binocular camera to a camera coordinate system of a virtual binocular camera, provided by some exemplary embodiments of the present disclosure;

[0015] Figure 4 is a schematic diagram of a process for determining external parameter information of a virtual binocular camera provided by some exemplary embodiments of the present disclosure;

[0016] Figure 5 is a schematic diagram of a process for determining a conversion matrix from a camera coordinate system of a real binocular camera to a camera coordinate system of a virtual binocular camera, provided by some other exemplary embodiments of the present disclosure;

[0017] Figure 6 is a schematic diagram of a process for obtaining normalized plane coordinates of a target object in a camera coordinate system of a virtual binocular camera, provided by some exemplary embodiments of the present disclosure;

[0018] Figure 7 is a schematic diagram of a process for obtaining the three-dimensional coordinates of a target object in a camera coordinate system of a virtual binocular camera provided by some exemplary embodiments of the present disclosure;

[0019] Figure 8 is a flowchart of a method for determining three-dimensional key points provided by some exemplary embodiments of the present disclosure;

[0020] Figure 9 is a schematic structural diagram of an apparatus for determining three-dimensional key points provided by some exemplary embodiments of the present disclosure;

[0021] Figure 10 is a schematic structural diagram of an apparatus for determining three-dimensional key points provided by other exemplary embodiments of the present disclosure;

[0022] Figure 11 It is a structural diagram of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0023] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0024] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0025] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0026] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0027] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0028] The embodiments of the present disclosure can be applied to electronic devices such as head-mounted display devices, terminal devices, servers, etc., which can operate with many other general-purpose or special-purpose computing system environments or configurations. Electronic devices such as head-mounted display devices, terminal devices, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, in which tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0029] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0030] Figure 1This is an exemplary system architecture that can be applied to some embodiments of the method or apparatus for determining three-dimensional key points disclosed herein. The method for determining three-dimensional key points disclosed herein can be applied to various electronic devices with image processing functions, such as mobile phones, tablets, computers, car computers, servers, head-mounted display devices, and other terminal devices. Figure 1 As shown, the system architecture may include an image acquisition device 1, a network 2, and an electronic device 3. The network 2 may be a medium for providing a communication link between the image acquisition device 1 and the electronic device 3. The network 2 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0031] The above-mentioned image acquisition device 1 can be any electronic device with image acquisition function, including but not limited to mobile phone cameras, tablet cameras, cameras installed on vehicles and connected to the vehicle computer, cameras of head-mounted display devices (such as AR glasses, VR glasses, etc.), etc.

[0032] In addition, the method for determining three-dimensional key points disclosed herein can also be applied to various electronic devices that have both image acquisition and image processing functions (for example, head-mounted display devices that have both camera and image processing functions).

[0033] The electronic device 3 can provide various services. For example, it can obtain a binocular image containing a target object captured by a real binocular camera; determine the two-dimensional coordinates of the target object in the binocular image; convert the two-dimensional coordinates of the target object in the binocular image to the normalized planes of the corresponding virtual binocular camera based on the conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, thereby obtaining the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera; wherein the virtual binocular camera is a parallel binocular camera; obtain the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and the three-dimensional key point prediction model of the virtual standard binocular camera; wherein the virtual standard binocular camera is a parallel binocular camera with a preset baseline length; and determine the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera based on the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0034] Exemplary Methods

[0035] Figure 2 1 is a flow chart of a method for determining three-dimensional key points provided by some exemplary embodiments of the present disclosure. The embodiments of the present disclosure can be applied to electronic devices, such as Figure 2 As shown, the method may include the following steps:

[0036] Step 210: Acquire a binocular image containing the target object captured by a real binocular camera.

[0037] In some optional embodiments of the present disclosure, a real binocular camera may refer to a binocular camera installed on a terminal device, which may be a head-mounted display device, a mobile phone, a tablet, etc.

[0038] In some optional embodiments of the present disclosure, the binocular image may include a first image and a second image.

[0039] In some optional embodiments of the present disclosure, the first image and the second image can be any of the following combinations: the first image is a left-eye image and the second image is a right-eye image; the first image is a right-eye image and the second image is a left-eye image. Optionally, the first image and the second image have the same resolution; or the first image and the second image have different resolutions.

[0040] In some optional embodiments of the present disclosure, the target object can be any object requiring 3D key point recognition, such as a human body, hand, face, animal, or vehicle. The human body, face, and hand represent different research areas. For example, the human body focuses on the overall shape of the human body at a large granularity, without focusing on facial expressions or fine hand movements. In contrast, the face focuses on fine facial expressions, and the hand focuses on hand gestures.

[0041] Step 220: Determine the two-dimensional coordinates of the target object in the binocular image.

[0042] In some optional embodiments of the present disclosure, the two-dimensional coordinates of the target object in the binocular image may be the two-dimensional coordinates of the key points of the target object in the binocular image. The key points of the target object can be set based on the actual situation of the target object. For example, if the target object is a human body, N key points can be set, such as head key points, neck key points, torso key points, arm key points, leg key points, etc. The key points of the human body can be used to represent major joints or movable ends of the human body. The key points can be used to accurately determine the shape of the target object. The location and number of key points on each human body part can be set according to actual needs. For example, the key points on the arm can include three key points: the shoulder key point, the elbow key point, and the hand key point, or two key points: the shoulder key point and the hand key point, without limitation. If the target object is a hand, the key points of the hand can be points used to represent gestures, such as points on the joints, fingertips, etc. If the target object is a human face, the key points of the human face can be points related to facial expression, such as the eyebrows, the center of the eyebrows, the end of the eyebrows, the corners of the eyes, the pupils, the tip of the nose, the corners of the mouth, the center of the upper lip, the center of the lower lip, etc. If the target object is an animal, the key points of the animal can be points used to represent the animal's posture, such as the eyes, nose, throat, ear base, tail, elbows, knees, paws, etc.

[0043] In some optional embodiments of the present disclosure, the two-dimensional coordinates of the target object in the binocular image may include first two-dimensional coordinates of the target object in the first image and second two-dimensional coordinates of the target object in the second image.

[0044] In some optional embodiments of the present disclosure, the two-dimensional coordinates of the target object in the binocular image can be determined using any feasible method. For example, any feasible two-dimensional key point detection network can be used to perform key point detection on the binocular image to obtain the two-dimensional coordinates of the target object in the binocular image.

[0045] In some optional embodiments of the present disclosure, target detection can be performed on the binocular image first to detect a detection frame of the target object in the binocular image, and then the target object area can be cropped from the binocular image according to the detection frame. Then, two-dimensional key point detection can be performed based on the cropped target object area to obtain the two-dimensional coordinates of the target object in the binocular image.

[0046] In some optional embodiments of the present disclosure, if the binocular image is a distorted image, the binocular image can be further dedistorted to obtain a dedistorted binocular image, and then based on the dedistorted binocular image, the two-dimensional coordinates of the target object in the dedistorted binocular image can be determined.

[0047] In some optional embodiments of the present disclosure, if the binocular image is a distorted image, after obtaining the two-dimensional coordinates of the target object in the binocular image, a dedistortion operation can be performed on the two-dimensional coordinates of the target object in the binocular image to obtain the dedistorted two-dimensional coordinates for determining the three-dimensional coordinates of the target object.

[0048] Step 230 : Based on the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the two-dimensional coordinates of the target object in the binocular image are transformed into the normalized plane of the corresponding virtual binocular camera to obtain the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0049] In some optional embodiments of the present disclosure, the virtual binocular camera may be a parallel binocular camera. The virtual binocular camera may include a first virtual camera and a second virtual camera. A parallel binocular camera may refer to a binocular camera in which the imaging planes of the two cameras are parallel and the epipolar lines of the two cameras coincide with the baseline.

[0050] In some optional embodiments of the present disclosure, a virtual binocular camera may refer to a virtual camera that is configured with relevant parameters of a binocular camera but does not have a real camera.

[0051] In some optional embodiments of the present disclosure, a transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera can be obtained using the extrinsic parameter information of the real binocular camera and the extrinsic parameter information of the virtual binocular camera. The transformation matrix may include at least one of a rotation matrix and a translation vector.

[0052] In some optional embodiments of the present disclosure, the image coordinate system of the binocular image can be converted to the camera coordinate system of the real binocular camera based on the intrinsic parameter information of the real binocular camera. Furthermore, the camera coordinate system of the real binocular camera can be converted to the camera coordinate system of the virtual binocular camera based on the conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera. Then, the camera coordinate system of the virtual binocular camera can be converted to the normalized plane based on the conversion relationship between the camera coordinate system of the virtual binocular camera and the normalized plane. Based on the above process, the two-dimensional coordinates of the target object in the binocular image can be converted to the corresponding normalized plane of the virtual binocular camera, and the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera can be obtained.

[0053] Step 240 , obtaining the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and the three-dimensional key point prediction model of the virtual standard binocular camera; wherein the virtual standard binocular camera is a parallel binocular camera with a preset baseline length.

[0054] In some optional embodiments of the present disclosure, the normalized plane coordinates and depth of the target object in the camera coordinate system of the virtual binocular camera can be combined to obtain the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera. The three-dimensional key point prediction model of the virtual standard binocular camera can be used to predict the depth of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera. For example, the three-dimensional key point prediction model can be expressed as Among them, v represents the virtual binocular camera, norm represents normalization, and θ represents the network parameters. and represents the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, f is the three-dimensional key point depth prediction model, Represents the depth in the camera coordinate system of the virtual binocular camera. Then the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera are It can be expressed as:

[0055] In some optional embodiments of the present disclosure, the 3D key point prediction model of the virtual standard binocular camera can also be implemented using any feasible 3D key point prediction network. For example, the 3D key point prediction model can be expressed as Among them, v represents the virtual binocular camera, norm represents normalization, and represents the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, f 3d is a 3D key point prediction model, represents the 3D coordinates of the target object in the virtual binocular camera's camera coordinate system. That is, the normalized planar coordinates of the target object in the virtual binocular camera's camera coordinate system serve as the input to the 3D key point prediction model of the virtual standard binocular camera. Through inference using the 3D key point prediction model, the 3D coordinates of the target object in the virtual binocular camera's camera coordinate system are obtained.

[0056] Step 250 : determining the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera according to the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0057] In some optional embodiments of the present disclosure, after obtaining the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera can be determined based on the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera. For example, based on a transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera can be transformed into the camera coordinate system of the real binocular camera to obtain the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera.

[0058] The method for determining three-dimensional key points provided by the embodiment of the present disclosure can determine the two-dimensional coordinates of the target object in the binocular image after obtaining a binocular image containing the target object captured by a real binocular camera. Then, based on the conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the two-dimensional coordinates of the target object in the binocular image can be converted to the normalized plane of the corresponding virtual binocular camera, and the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera can be obtained. Then, based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and the three-dimensional key point prediction model of the virtual standard binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera can be obtained, so that the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera can be determined based on the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera. Since the virtual binocular camera is a parallel binocular camera, by converting the two-dimensional coordinates of the target object in the binocular image captured by the real binocular camera to the normalized plane of the corresponding virtual binocular camera, it is convenient to estimate the three-dimensional key points under the virtual binocular camera through a unified virtual standard three-dimensional key point prediction model, thereby helping to achieve that real binocular cameras with different arrangements can use the same standard three-dimensional key point prediction model for three-dimensional key point estimation. There is no need to use different data sets for real binocular cameras with different arrangements to jointly train the three-dimensional key point prediction model, which greatly reduces the workload of collecting training data sets and improves model training efficiency.

[0059] Figure 3 The present invention is a flowchart of determining a conversion matrix from a camera coordinate system of a real binocular camera to a camera coordinate system of a virtual binocular camera, provided by some exemplary embodiments of the present disclosure.

[0060] In some optional embodiments of the present disclosure, such as Figure 3 As shown, before the step of converting the two-dimensional coordinates of the target object in the binocular image to the normalized plane of the corresponding virtual binocular camera based on the conversion matrix between the camera coordinate system of the real binocular camera and the camera coordinate system of the virtual binocular camera, and obtaining the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, the method of the embodiment of the present disclosure may further include:

[0061] Step 310 : Determine a vector connecting an optical center of a first camera and an optical center of a second camera of the real binocular camera based on extrinsic parameter information of the real binocular camera.

[0062] In some optional embodiments of the present disclosure, the first camera and the second camera can be any of the following combinations: the first camera is the left camera of a real binocular camera, and the second camera is the right camera of a real binocular camera; the first camera is the right camera of a real binocular camera, and the second camera is the left camera of a real binocular camera.

[0063] In some optional embodiments of the present disclosure, the external parameter information of the real binocular camera may include the pose information of the real binocular camera in the world coordinate system, which can be expressed as r represents a real binocular camera; the world coordinate system can be arbitrary and can be freely defined as needed. For example, the camera coordinate system of any camera in the first camera and the second camera can be used as the world coordinate system. The pose information can include a rotation matrix (for example, it can be expressed as ) and the translation vector (which can be expressed as ) in at least one of .

[0064] In some optional embodiments of the present disclosure, the vector connecting the optical center of the first camera and the optical center of the second camera can be obtained by the coordinates of the optical center of the first camera and the optical center of the second camera in the world coordinate system. For example, if the world coordinate system is the camera coordinate system of the first camera, then the vector connecting the optical center of the first camera and the optical center of the second camera is the translation vector in the pose information of the second camera.

[0065] In step 320 , the vector is used as the direction of the baseline of the virtual binocular camera to determine the external parameter information of the virtual binocular camera.

[0066] In some optional embodiments of the present disclosure, the external parameter information of the virtual binocular camera may include the pose information of the virtual binocular camera in the above-mentioned world coordinate system, which can be expressed as The pose information may include the rotation matrix of the virtual binocular camera (for example, it can be expressed as ) and the translation vector (which can be expressed as ) in at least one of .

[0067] In some optional embodiments of the present disclosure, the optical center of the first camera of the real binocular camera can be aligned with the optical center position of the first virtual camera of the virtual binocular camera to obtain the position of the optical center of the first virtual camera in the world coordinate system. Using the above vector as the direction of the baseline of the virtual binocular camera, the position of the optical center of the second virtual camera of the virtual binocular camera in the world coordinate system can be obtained. Combined with other relevant parameters of the virtual binocular camera, the external parameter information of the virtual binocular camera is obtained.

[0068] Step 330 : Determine a conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera based on the extrinsic parameter information of the real binocular camera and the extrinsic parameter information of the virtual binocular camera.

[0069] In some optional embodiments of the present disclosure, after obtaining the extrinsic parameter information of the virtual binocular camera, a transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera can be determined based on the extrinsic parameter information of the real binocular camera and the extrinsic parameter information of the virtual binocular camera. In other words, a rotation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera can be calculated.

[0070] The disclosed embodiments can determine the vector connecting the optical center of the first camera and the optical center of the second camera of the real binocular camera, and use the vector as the direction of the baseline of the virtual binocular camera. This allows the baseline of the virtual binocular camera to coincide with the baseline of the real binocular camera, facilitating the determination of the baseline of the virtual binocular camera. This allows the accurate determination of the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera.

[0071] In some optional embodiments of the present disclosure, the baseline of the real binocular camera 1 is b1, and the baseline of the real binocular camera 2 is b2; after the real binocular camera 1 is rotated into a parallel binocular camera, the virtual binocular camera 1 is obtained; after the real binocular camera 2 is rotated into a parallel binocular camera, the virtual binocular camera 2 is obtained. For example, the optical center of the first camera of the real binocular camera 1 is O1, and the optical center of the second camera is O2. The vector As the direction of the baseline of the virtual binocular camera 1, the rotation matrix from the real binocular camera 1 to the virtual binocular camera 1 is obtained, so that the camera coordinate system of the real binocular camera 1 can meet the conditions of parallel binocular cameras after rotating according to the rotation matrix. Similarly, for the real binocular camera 2, the optical center of the first camera of the real binocular camera 2 is O3, and the optical center of the second camera is O4. The direction of the baseline of the virtual binocular camera 2 is used to obtain the rotation matrix from the real binocular camera 2 to the virtual binocular camera 2, and the details are not repeated here.

[0072] In some embodiments of the present disclosure, the baseline of the virtual binocular camera can be determined based on the external parameters of the virtual binocular camera in the world coordinate system. For example, the baseline b1 of the virtual binocular camera 1 can be obtained based on the external parameters of the virtual binocular camera 1 in the world coordinate system; the baseline b2 of the virtual binocular camera 2 can be obtained based on the external parameters of the virtual binocular camera 2 in the world coordinate system.

[0073] Figure 4 It is a flowchart of determining the external parameter information of a virtual binocular camera provided by some exemplary embodiments of the present disclosure.

[0074] In some optional embodiments of the present disclosure, such as Figure 4 As shown, the vector is used as the direction of the baseline of the virtual binocular camera to determine the external parameter information of the virtual binocular camera, which may include:

[0075] Step 3210 : Determine a first rotation matrix from the transverse axis of the first camera to the vector, and a second rotation matrix from the transverse axis of the second camera to the vector.

[0076] In some optional embodiments of the present disclosure, the horizontal axis of the first camera is the X axis of the first camera. The optical axis direction of the camera is the Z axis direction, the direction perpendicular to the Z axis is the Y axis direction, and the direction perpendicular to the YOZ axis is the X axis direction.

[0077] In some optional embodiments of the present disclosure, taking a real binocular camera 1 as an example, the vector is The horizontal axis of the first camera is X1, and the horizontal axis of the second camera is X2. The camera coordinate system of the first camera is used as the world coordinate system. According to the position and vector of the horizontal axis X1 of the first camera in the world coordinate system In the world coordinate system, the position of the first camera's horizontal axis X1 to the vector Similarly, we can get the first rotation matrix of the second camera's horizontal axis X2 to the vector The second rotation matrix.

[0078] Step 3220: Determine the extrinsic parameter information of the virtual binocular camera based on the extrinsic parameter information of the real binocular camera, the first rotation matrix, and the second rotation matrix.

[0079] In some optional embodiments of the present disclosure, the extrinsic parameter information of the first camera and the second camera of the real binocular camera are rotated according to the first rotation matrix and the second rotation matrix respectively to obtain the extrinsic parameter information of the virtual binocular camera.

[0080] The disclosed embodiments use the vector connecting the optical center of the first camera and the optical center of the second camera of a real binocular camera as a rotation reference to calculate a first rotation matrix from the transverse axis of the first camera to the vector and a second rotation matrix from the transverse axis of the second camera to the vector. This ensures that the rotated first and second cameras meet the requirements of a virtual binocular camera. Therefore, the first and second rotation matrices can represent the transformation relationship between the real binocular camera and the virtual binocular camera. Based on the extrinsic parameter information of the real binocular camera, the first and second rotation matrices, the extrinsic parameter information of the virtual binocular camera can be effectively obtained, facilitating the conversion of the two-dimensional coordinates of the target object in the binocular image to the camera coordinate system of the virtual binocular camera, thereby enabling the prediction of the three-dimensional coordinates of the target object using a standard three-dimensional key point prediction model.

[0081] Figure 5 This is a schematic diagram of a process for determining a conversion matrix from a camera coordinate system of a real binocular camera to a camera coordinate system of a virtual binocular camera, provided by some other exemplary embodiments of the present disclosure.

[0082] In some optional embodiments of the present disclosure, such as Figure 5As shown, based on the extrinsic parameter information of the real binocular camera and the extrinsic parameter information of the virtual binocular camera, determining the conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera may include:

[0083] Step 3310 : Determine a conversion matrix from the camera coordinate system of the first camera to the camera coordinate system of the first virtual camera based on the extrinsic parameter information of the first camera and the extrinsic parameter information of the first virtual camera of the virtual binocular camera.

[0084] In some optional embodiments of the present disclosure, the external parameter information of the first camera in the world coordinate system can be expressed as The external parameter information of the first virtual camera in the world coordinate system can be expressed as Then the transformation matrix from the camera coordinate system of the first camera to the camera coordinate system of the first virtual camera can be expressed as: Express request The inverse matrix of .

[0085] Step 3320 : Determine a conversion matrix from the camera coordinate system of the second camera to the camera coordinate system of the second virtual camera based on the extrinsic parameter information of the second camera and the extrinsic parameter information of the second virtual camera of the virtual binocular camera.

[0086] In some optional embodiments of the present disclosure, the external parameter information of the second camera in the world coordinate system can be expressed as The external parameter information of the second virtual camera in the world coordinate system can be expressed as Then the transformation matrix from the camera coordinate system of the second camera to the camera coordinate system of the second virtual camera can be expressed as: Express request The inverse matrix of .

[0087] In some optional embodiments of the present disclosure, the calculation formula of the transformation matrix can be derived in the following way: Let a point in the world coordinate system be P W The coordinates of this point in the camera coordinate system of the real binocular camera (for example, the camera coordinate system of the first camera) are expressed as P C The coordinates of this point in the camera coordinate system of the virtual binocular camera (for example, the camera coordinate system of the first virtual camera) are expressed as P v , then the following relationship exists:

[0088]

[0089] in, Represents the transformation matrix from the camera coordinate system of the first camera to the world coordinate system, Representing the transformation matrix from the camera coordinate system of the first virtual camera to the world coordinate system, we can get:

[0090]

[0091] Therefore, the transformation matrix from the camera coordinate system of the first camera to the camera coordinate system of the first virtual camera is

[0092] In some optional embodiments, the baseline b of the virtual binocular camera v It can be expressed as:

[0093]

[0094] The disclosed embodiment obtains a conversion matrix from the camera coordinate system of the first camera to the camera coordinate system of the first virtual camera through the extrinsic parameter information of the first camera and the extrinsic parameter information of the first virtual camera, and obtains a conversion matrix from the camera coordinate system of the second camera to the camera coordinate system of the second virtual camera through the extrinsic parameter information of the second camera and the extrinsic parameter information of the second virtual camera, thereby obtaining a conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, which helps to convert the two-dimensional coordinates of the target object in the binocular image to the normalized plane of the corresponding virtual binocular camera, so as to predict the three-dimensional coordinates of the target object through a standard three-dimensional key point prediction model.

[0095] Figure 6 It is a schematic diagram of a process for obtaining the normalized plane coordinates of a target object in the camera coordinate system of a virtual binocular camera provided by some exemplary embodiments of the present disclosure.

[0096] In some optional embodiments of the present disclosure, the two-dimensional coordinates of the target object in the binocular image include first two-dimensional coordinates corresponding to the first image and second two-dimensional coordinates corresponding to the second image in the binocular image.

[0097] In some optional embodiments of the present disclosure, such as Figure 6 As shown, based on the conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the two-dimensional coordinates of the target object in the binocular image are converted to the normalized plane of the corresponding virtual binocular camera, and the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera are obtained, which may include:

[0098] Step 2310 : Project the two-dimensional coordinates of the target object in the binocular image onto normalized planes in the camera coordinate system of the real binocular camera to obtain normalized plane projection coordinates of the target object in the camera coordinate system of the real binocular camera.

[0099] In some optional embodiments of the present disclosure, the two-dimensional coordinates of the target object in the binocular image can be projected onto the normalized plane in the camera coordinate system of the real binocular camera according to the conversion relationship between the image coordinate system of the binocular image and the normalized plane in the camera coordinate system of the real binocular camera, thereby obtaining the normalized plane projection coordinates of the target object in the camera coordinate system of the real binocular camera. The conversion relationship between the image coordinate system of the binocular image and the normalized plane in the camera coordinate system of the real binocular camera can be obtained in advance. For example, the conversion relationship between the image coordinate system of the binocular image and the camera coordinate system of the real binocular camera can be obtained according to the intrinsic parameters of the real binocular camera, and the normalized plane is the Z=1 plane in the camera coordinate system. Based on this, the two-dimensional coordinates of the target object in the binocular image can be projected onto the normalized plane in the camera coordinate system of the real binocular camera, thereby obtaining the normalized plane projection coordinates of the target object in the camera coordinate system of the real binocular camera, which can be expressed as norm means normalization.

[0100] In some optional embodiments of the present disclosure, the first two-dimensional coordinates can be projected onto the normalized plane of the camera coordinate system of the first camera of the real binocular camera to obtain the normalized plane projection coordinates of the target object in the camera coordinate system of the first camera, which can be expressed as The second two-dimensional coordinate can be projected onto the normalized plane of the camera coordinate system of the second camera to obtain the normalized plane projection coordinates of the target object in the camera coordinate system of the second camera, for example, it can be expressed as

[0101] Step 2320: Based on the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the normalized plane projection coordinates are transformed into the normalized planes in the camera coordinate system of the corresponding virtual binocular camera to obtain the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0102] In some optional embodiments of the present disclosure, the normalized plane projection coordinates can be converted to the corresponding camera coordinate system of the virtual binocular camera based on the conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, thereby obtaining the coordinates of the normalized plane projection coordinates in the camera coordinate system of the virtual binocular camera. The coordinates of the normalized plane projection coordinates in the camera coordinate system of the virtual binocular camera can then be projected onto the normalized plane of the virtual binocular camera to obtain the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera. For example, the coordinates of the normalized plane projection coordinates in the camera coordinate system of the virtual binocular camera can be divided by z to obtain the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera. z represents the z coordinate of the normalized plane projection coordinates in the camera coordinate system of the virtual binocular camera.

[0103] In some optional embodiments of the present disclosure, the normalized plane projection coordinates of the target object on the normalized plane of the first camera may be Convert to the normalized plane of the first virtual camera of the virtual binocular camera to obtain the first normalized plane coordinates of the target object in the normalized plane of the first virtual camera, which can be expressed as The target object can be projected onto the normalized plane of the second camera using the normalized plane projection coordinates Convert to the normalized plane of the second virtual camera of the virtual binocular camera to obtain the second normalized plane coordinates of the target object in the normalized plane of the second virtual camera, which can be expressed as

[0104] The disclosed embodiment projects the two-dimensional coordinates of the target object in the binocular image onto the normalized plane in the camera coordinate system of the real binocular camera, thereby obtaining the normalized plane projection coordinates of the target object in the camera coordinate system of the real binocular camera. The normalized projection coordinates are then converted to the normalized planes of the corresponding virtual binocular camera, thereby effectively obtaining the normalized plane coordinates of the target object in the virtual binocular camera, thereby facilitating the prediction of the three-dimensional coordinates of the target object using a standard three-dimensional key point prediction model.

[0105] Figure 7 It is a schematic diagram of a process for obtaining the three-dimensional coordinates of a target object in the camera coordinate system of a virtual binocular camera provided by some exemplary embodiments of the present disclosure.

[0106] In some optional embodiments of the present disclosure, such as Figure 7 As shown, according to the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and the three-dimensional key point prediction model of the virtual standard binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera are obtained, which may include:

[0107] Step 2410 : Determine a baseline scale factor based on the baseline length of the virtual binocular camera and a preset baseline length of the virtual standard binocular camera.

[0108] In some optional embodiments of the present disclosure, the baseline length of the virtual binocular camera 1 is obtained as b1 according to the external parameters of the virtual binocular camera 1 in the world coordinate system, and the baseline length of the virtual binocular camera 2 is obtained as b2 according to the external parameters of the virtual binocular camera 2 in the world coordinate system. The preset baseline length of the virtual standard binocular camera is b t , then the baseline scale factor of the virtual binocular camera 1 can be determined as scale1 = b1 / b t , the baseline scale factor of virtual binocular camera 2 scale2 = b2 / b t .

[0109] Step 2420 : Determine the depth of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, the 3D key point prediction model of the virtual standard binocular camera, and the baseline scale factor.

[0110] In some optional embodiments of the present disclosure, the 3D key point prediction model of the virtual standard binocular camera can be a pre-trained 3D key point prediction model. The input of the 3D key point prediction model is the normalized plane coordinates of the object (which can be the target object or other objects) in the camera coordinate system of the virtual binocular camera, and the output is the predicted depth of the object in the camera coordinate system of the virtual standard binocular camera, which can be expressed as For example, a batch of training binocular images can be obtained in advance, and the two-dimensional coordinates of the objects in the training binocular images can be converted to the normalized plane of the virtual binocular camera. The normalized plane coordinates of the objects in the normalized plane of the virtual binocular camera are obtained as training samples for training the three-dimensional key point prediction model. The three-dimensional key point prediction model of the virtual standard binocular camera is obtained through training.

[0111] According to the relationship between binocular camera depth, parallax and baseline (z represents depth, d represents parallax, f represents focal length, and b represents the baseline length of the binocular camera) It can be seen that the depth is proportional to the baseline length. The baseline length b of the virtual binocular camera is the same as the preset baseline length b of the virtual standard binocular camera. t Has a certain ratio, expressed as the baseline scale factor Then the depth of the target object in the camera coordinate system of the virtual binocular camera and the depth of the target object in the camera coordinate system of the virtual standard binocular camera also have an equal ratio.

[0112] In some optional embodiments of the present disclosure, the depth of the target object in the camera coordinate system of the virtual standard binocular camera is used. And the baseline scale factor scale can determine the depth of the target object in the camera coordinate system of the virtual binocular camera. For example, the depth of the target object in the camera coordinate system of the virtual binocular camera 1 is The depth of the target object in the camera coordinate system of the virtual binocular camera 1 is

[0113] Step 2430 : Obtain the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera based on the depth of the target object in the camera coordinate system of the virtual binocular camera and the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0114] In some optional embodiments of the present disclosure, the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera can be expressed as: Indicates the depth of the target object in the camera coordinate system of the virtual binocular camera, Represents the normalized plane coordinates of the target object in the normalized plane of the first virtual camera.

[0115] In some optional embodiments of the present disclosure, the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera can also be expressed as: Represents the normalized plane coordinates of the target object in the normalized plane of the second virtual camera.

[0116] The disclosed embodiment determines the depth of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, the three-dimensional key point prediction model of the virtual standard binocular camera, and the baseline scale factor. Then, based on the depth of the target object in the camera coordinate system of the virtual binocular camera and the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera are obtained. Thus, the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera are effectively obtained based on the virtual standard three-dimensional key point prediction model, so that binocular images captured by various different real binocular cameras can realize the prediction of the three-dimensional coordinates of the object through a unified standard three-dimensional key point prediction model.

[0117] In some optional embodiments of the present disclosure, determining the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera based on the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera may include: determining the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera based on the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, and the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0118] In some optional embodiments of the present disclosure, the three-dimensional key point of the virtual binocular camera is the depth obtained under the virtual binocular camera. Therefore, based on the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera can be transformed into the camera coordinate system of the real binocular camera to obtain the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera.

[0119] In some optional embodiments of the present disclosure, the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera are It can be expressed as: R left Represents the transformation matrix from the camera coordinate system of the first camera to the camera coordinate system of the first virtual camera. Represents the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0120] In some optional embodiments of the present disclosure, Figure 8 FIG is a flowchart of a method for determining three-dimensional key points provided by some exemplary embodiments of the present disclosure. Figure 8 As shown, according to the external parameter information of the real binocular camera, the conversion matrix R from the left camera of the real binocular camera to the left virtual camera of the virtual binocular camera can be determined in the above manner. left And the transformation matrix R from the right camera of the real binocular camera to the right virtual camera of the virtual binocular camera right According to the left and right purpose transformation matrix and the normalized projection coordinates of the target object on the normalized plane of the real binocular camera (include and ), the normalized plane coordinates of the target object in the normalized plane of the virtual binocular camera can be obtained. The normalized plane coordinates of the target object in the normalized plane of the virtual binocular camera are input into the 3D key point prediction model of the virtual standard binocular camera to obtain the depth of the target object in the camera coordinate system of the virtual standard binocular camera. Combined with the baseline scale factor scale and the depth of the target object in the camera coordinate system of the virtual binocular camera The depth of the target object in the camera coordinate system of the virtual binocular camera can be calculated Then we can combine the depth of the target object in the camera coordinate system of the virtual binocular camera Normalized projection coordinates of the target object on the normalized plane of the virtual binocular camera And the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera can be obtained based on Figure 8 The process shown in the figure can be used to predict 3D coordinates for any arrangement of real binocular cameras using a unified standard 3D key point prediction model.

[0121] In related technologies, epipolar correction is typically used to align the epipolar lines and baselines of binocular cameras. Specifically, the camera coordinate system of the left camera is set to the world coordinate system, and the extrinsic parameters of the right camera include the rotation matrix R and the translation vector t. The left and right cameras are each rotated by half of the rotation matrix R, so that the imaging planes of the two cameras are parallel and the epipolar lines of the two cameras coincide with the baselines. However, this epipolar correction method can only achieve ideal parallel binocularity for one pair of binocular cameras. Binocular cameras with different layouts cannot be unified into a fixed standard virtual binocular camera. Therefore, it is still necessary to use different data sets for joint training to obtain a 3D key point prediction model. The model training workload is large and the efficiency is low.

[0122] The method for determining 3D key points provided by the disclosed embodiments can convert real binocular cameras of varying configurations into virtual parallel binocular cameras. By training a 3D key point prediction model for a virtual standard parallel binocular camera, the method can be applied to predict the 3D coordinates of target objects for real binocular cameras of varying configurations. For example, the method can be applied to smart glasses of varying configurations to accommodate different models and assembly errors. Therefore, the method of the disclosed embodiments can meet the requirements of standard binocular cameras, significantly reducing the model training workload and improving training efficiency.

[0123] The above-mentioned embodiments or optional examples of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0124] Any of the methods for determining three-dimensional key points provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, any of the methods for determining three-dimensional key points provided in the embodiments of the present disclosure can be executed by a processor, such as by invoking corresponding instructions stored in a memory to execute any of the methods for determining three-dimensional key points mentioned in the embodiments of the present disclosure. This will not be further described below.

[0125] Exemplary devices

[0126] Figure 9 Schematic diagram of the structure of the device for determining three-dimensional key points provided by some exemplary embodiments of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiments of the present disclosure, such as Figure 9 The illustrated apparatus may include: an image acquisition module 410 , a two-dimensional coordinate determination module 420 , a normalization module 430 , a virtual three-dimensional coordinate determination module 440 , and a real three-dimensional coordinate determination module 450 .

[0127] The image acquisition module 410 is used to acquire a binocular image containing a target object captured by a real binocular camera.

[0128] The two-dimensional coordinate determination module 420 is configured to determine the two-dimensional coordinates of the target object in the binocular image.

[0129] The normalization module 430 is configured to convert the two-dimensional coordinates of the target object in the binocular image to the corresponding normalized planes of the virtual binocular camera based on the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, thereby obtaining the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera. The virtual binocular camera is a parallel binocular camera.

[0130] The virtual three-dimensional coordinate determination module 440 is configured to obtain the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and the three-dimensional key point prediction model of the virtual standard binocular camera. The virtual standard binocular camera is a parallel binocular camera with a preset baseline length.

[0131] The real three-dimensional coordinate determination module 450 is configured to determine the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera according to the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0132] Figure 10 3D key point determination apparatus according to some other exemplary embodiments of the present disclosure.

[0133] In some optional embodiments of the present disclosure, such as Figure 10 As shown, the device of the embodiment of the present disclosure may further include:

[0134] The vector determination module 510 may be configured to determine a vector connecting an optical center of a first camera and an optical center of a second camera of the real binocular camera based on external parameter information of the real binocular camera.

[0135] The extrinsic parameter determination module 520 may be configured to determine extrinsic parameter information of the virtual binocular camera using a vector as the direction of the baseline of the virtual binocular camera.

[0136] The transformation matrix determination module 530 may be configured to determine a transformation matrix from a camera coordinate system of the real binocular camera to a camera coordinate system of the virtual binocular camera based on extrinsic parameter information of the real binocular camera and extrinsic parameter information of the virtual binocular camera.

[0137] In some optional embodiments of the present disclosure, the extrinsic parameter determination module 520 may be specifically configured to determine a first rotation matrix from the transverse axis of the first camera to the vector, and a second rotation matrix from the transverse axis of the second camera to the vector. Based on the extrinsic parameter information of the real binocular camera, the first rotation matrix, and the second rotation matrix, the extrinsic parameter information of the virtual binocular camera is determined.

[0138] In some optional embodiments of the present disclosure, the transformation matrix determination module 530 may be specifically configured to: determine a transformation matrix from the camera coordinate system of the first camera to the camera coordinate system of the first virtual camera based on the extrinsic parameter information of the first camera and the extrinsic parameter information of the first virtual camera of the virtual binocular camera; and determine a transformation matrix from the camera coordinate system of the second camera to the camera coordinate system of the second virtual camera based on the extrinsic parameter information of the second camera and the extrinsic parameter information of the second virtual camera of the virtual binocular camera.

[0139] In some optional embodiments of the present disclosure, the two-dimensional coordinates of the target object in the binocular image include first two-dimensional coordinates corresponding to the first image and second two-dimensional coordinates corresponding to the second image in the binocular image.

[0140] In some optional embodiments of the present disclosure, the normalization module 430 may be specifically configured to project the two-dimensional coordinates of the target object in the binocular image onto normalized planes in the camera coordinate system of the real binocular camera to obtain the normalized plane projection coordinates of the target object in the camera coordinate system of the real binocular camera. Based on a transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the normalized plane projection coordinates are transformed into normalized planes in the camera coordinate system of the corresponding virtual binocular camera to obtain the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0141] In some optional embodiments of the present disclosure, the virtual three-dimensional coordinate determination module 440 may be specifically configured to: determine a baseline scale factor based on the baseline length of the virtual binocular camera and a preset baseline length of the virtual standard binocular camera; determine the depth of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, a three-dimensional key point prediction model of the virtual standard binocular camera, and the baseline scale factor; and obtain the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera based on the depth of the target object in the camera coordinate system of the virtual binocular camera and the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0142] In some optional embodiments of the present disclosure, the real three-dimensional coordinate determination module 450 can be specifically used to determine the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera based on the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, and the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

[0143] It should be noted that the specific implementation of the device for determining three-dimensional key points in the embodiment of the present disclosure is similar to the specific implementation of the method for determining three-dimensional key points in the embodiment of the present disclosure. Please refer to the part of the method for determining three-dimensional key points for details. In order to reduce redundancy, they will not be described here.

[0144] Exemplary electronic devices

[0145] An embodiment of the present disclosure further provides an electronic device, including: a processor, and a memory for storing instructions executable by the processor.

[0146] The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method for determining three-dimensional key points described in any of the above embodiments of the present disclosure.

[0147] Exemplary head-mounted display device

[0148] An embodiment of the present disclosure further provides an electronic device, including: a processor, and a memory for storing instructions executable by the processor.

[0149] The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method for determining three-dimensional key points described in any of the above embodiments of the present disclosure.

[0150] Figure 11 1 is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. The structure of the head-mounted display device can refer to the structure of the electronic device. In this embodiment, the electronic device 100 includes one or more processors 110 and a memory 120.

[0151] The processor 110 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0152] The memory 120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 110 may execute the program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0153] In one example, the electronic device 100 may further include an input device 130 and an output device 140 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0154] For example, the input device 130 may be the aforementioned microphone or microphone array, configured to capture input signals from a sound source.

[0155] In addition, the input device 130 may also include, for example, a keyboard, a mouse, and the like.

[0156] The output device 140 can output various information to the outside, including determined distance information, direction information, etc. The output device 140 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0157] Of course, to simplify, Figure 11 Only some of the components related to the present disclosure in the electronic device 100 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device 100 may further include any other appropriate components according to specific application scenarios.

[0158] Exemplary computer program products and computer-readable storage media

[0159] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0160] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0161] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to execute the steps in the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0162] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0163] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0164] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For the electronic device embodiments, since they are generally similar to the method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0165] The block diagrams of the devices, apparatuses, and equipment involved in this disclosure are intended to be illustrative examples only and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and may be used interchangeably therewith. The words "or" and "" used herein refer to the words "and / or" and may be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and may be used interchangeably therewith.

[0166] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0167] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0168] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0169] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for determining three-dimensional key points, comprising: Obtain a binocular image containing the target object captured by a real binocular camera; Determining the two-dimensional coordinates of the target object in the binocular image; Based on the conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the two-dimensional coordinates of the target object in the binocular image are respectively converted to the corresponding normalized planes of the virtual binocular camera to obtain the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera; wherein the virtual binocular camera is a parallel binocular camera; Obtaining the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and the three-dimensional key point prediction model of the virtual standard binocular camera; wherein the virtual standard binocular camera is a parallel binocular camera with a preset baseline length; The three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera are determined according to the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

2. The method according to claim 1, wherein Before the step of converting the two-dimensional coordinates of the target object in the binocular image to the corresponding normalized planes of the virtual binocular camera based on the conversion matrix of the camera coordinate system of the real binocular camera and the camera coordinate system of the virtual binocular camera, and obtaining the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, the method further includes: Determining, based on extrinsic parameter information of the real binocular camera, a vector connecting an optical center of a first camera and an optical center of a second camera of the real binocular camera; Taking the vector as the direction of the baseline of the virtual binocular camera, determining the external parameter information of the virtual binocular camera; Based on the extrinsic parameter information of the real binocular camera and the extrinsic parameter information of the virtual binocular camera, a conversion matrix from a camera coordinate system of the real binocular camera to a camera coordinate system of the virtual binocular camera is determined.

3. The method according to claim 2, wherein: The step of determining the external parameter information of the virtual binocular camera by using the vector as the direction of the baseline of the virtual binocular camera includes: Determine a first rotation matrix from the transverse axis of the first camera to the vector, and a second rotation matrix from the transverse axis of the second camera to the vector; Based on the extrinsic parameter information of the real binocular camera, the first rotation matrix, and the second rotation matrix, the extrinsic parameter information of the virtual binocular camera is determined.

4. The method according to claim 2, wherein: The step of determining a conversion matrix from a camera coordinate system of the real binocular camera to a camera coordinate system of the virtual binocular camera based on the extrinsic parameter information of the real binocular camera and the extrinsic parameter information of the virtual binocular camera includes: Determining a transformation matrix from a camera coordinate system of the first camera to a camera coordinate system of the first virtual camera based on the extrinsic parameter information of the first camera and the extrinsic parameter information of the first virtual camera of the virtual binocular camera; Based on the extrinsic parameter information of the second camera and the extrinsic parameter information of the second virtual camera of the virtual binocular camera, a transformation matrix from the camera coordinate system of the second camera to the camera coordinate system of the second virtual camera is determined.

5. The method according to any one of claims 2 to 4, wherein: The two-dimensional coordinates of the target object in the binocular image include first two-dimensional coordinates corresponding to the first image and second two-dimensional coordinates corresponding to the second image in the binocular image; The method comprises: converting the two-dimensional coordinates of the target object in the binocular image to the corresponding normalized planes of the virtual binocular camera based on the conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, and obtaining the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, including: Projecting the two-dimensional coordinates of the target object in the binocular image onto normalized planes in the camera coordinate system of the real binocular camera, respectively, to obtain normalized plane projection coordinates of the target object in the camera coordinate system of the real binocular camera; Based on the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, the normalized plane projection coordinates are respectively transformed into the normalized planes in the corresponding camera coordinate system of the virtual binocular camera to obtain the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera.

6. The method according to any one of claims 2 to 5, wherein: The method of obtaining the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera according to the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and the three-dimensional key point prediction model of the virtual standard binocular camera includes: Determining a baseline scale factor based on the baseline length of the virtual binocular camera and a preset baseline length of the virtual standard binocular camera; Determining the depth of the target object in the camera coordinate system of the virtual binocular camera according to the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, the three-dimensional key point prediction model of the virtual standard binocular camera, and the baseline scale factor; Based on the depth of the target object in the camera coordinate system of the virtual binocular camera and the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera are obtained.

7. The method according to any one of claims 2 to 6, wherein: The determining, according to the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera comprises: Based on the transformation matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera and the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera, the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera are determined.

8. A device for determining three-dimensional key points, comprising: An image acquisition module is used to acquire a binocular image containing a target object captured by a real binocular camera; a two-dimensional coordinate determination module, configured to determine the two-dimensional coordinates of the target object in the binocular image; a normalization module, configured to convert the two-dimensional coordinates of the target object in the binocular image to the corresponding normalized planes of the virtual binocular camera based on a conversion matrix from the camera coordinate system of the real binocular camera to the camera coordinate system of the virtual binocular camera, thereby obtaining the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera; wherein the virtual binocular camera is a parallel binocular camera; a virtual three-dimensional coordinate determination module, configured to obtain the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera based on the normalized plane coordinates of the target object in the camera coordinate system of the virtual binocular camera and a three-dimensional key point prediction model of the virtual standard binocular camera; The real three-dimensional coordinate determination module is used to determine the three-dimensional coordinates of the target object in the camera coordinate system of the real binocular camera according to the three-dimensional coordinates of the target object in the camera coordinate system of the virtual binocular camera.

9. A computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 7.

11. A head-mounted display device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 7.

12. A computer program product, comprising computer program instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images