Camera pose calculation method, system and device based on multi-class image processing

Abstract

The application discloses a camera pose calculation method, system and device based on multi-class image processing, and the method comprises the following steps: obtaining a first image coordinate point corresponding to a first three-dimensional coordinate point from a two-dimensional image captured by a two-dimensional camera; obtaining a second image coordinate point corresponding to a second three-dimensional coordinate point from a depth image captured by a depth camera; projecting the first image coordinate point, the second image coordinate point, the first three-dimensional coordinate point and the second three-dimensional coordinate point onto a standardized spherical surface to obtain a first projection coordinate, a second projection coordinate, a third projection coordinate and a fourth projection coordinate respectively; and calculating the pose of the two-dimensional camera and the pose of the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate. The application unifies the pose calculation of the two-dimensional camera and the depth camera into the same calculation process, so that the calculation amount is small and the calculation efficiency is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a camera pose calculation method, system and device based on multi-class image processing. BACKGROUND

[0002] Currently, a plurality of cameras are usually arranged on a robot applying machine vision technology, for example, a visible light camera and a depth camera are arranged on the robot. Camera pose optimization of these cameras is very important for map construction, positioning and target tracking.

[0003] In the related art, the pose of each camera arranged on the robot is calculated based on the image and the three-dimensional point in the real scene photographed by each camera, respectively. For example, the pose of the visible light camera is calculated based on the image photographed by the visible light camera and the three-dimensional point in the real scene photographed by the visible light camera, and the pose of the depth camera is calculated based on the image photographed by the depth camera and the three-dimensional point in the real scene photographed by the depth camera. The pose calculation processes of the visible light camera and the depth camera are independent of each other and do not affect each other.

[0004] The above related art separately calculates the pose of each camera on the robot, which has large amount of calculation and low calculation efficiency. SUMMARY

[0005] To solve the above problems, the present application provides a camera pose calculation method, system and device based on multi-class image processing. The image coordinate points photographed by the two-dimensional camera, the image coordinate points photographed by the depth camera and the three-dimensional coordinate points in the real scene photographed are all projected onto the same standardized spherical surface, and the pose calculation of the two-dimensional camera and the depth camera is unified into the same calculation process, so that the poses of the two-dimensional camera and the depth camera can be calculated synchronously, which has small amount of calculation and high calculation efficiency.

[0006] In a first aspect, an embodiment of the present application provides a camera pose calculation method based on multi-class image processing, comprising:

[0007] obtaining a first image coordinate point corresponding to a first three-dimensional coordinate point in a real scene from a two-dimensional image photographed by a two-dimensional camera; and obtaining a second image coordinate point corresponding to a second three-dimensional coordinate point in the real scene from a depth image photographed by a depth camera;

[0008] projecting the first image coordinate point and the second image coordinate point onto a standardized spherical surface to obtain a first projection coordinate corresponding to the first image coordinate point and a second projection coordinate corresponding to the second image coordinate point, respectively; and the center of the standardized spherical surface is the optical center of the two-dimensional camera;

[0009] projecting the first three-dimensional coordinate point and the second three-dimensional coordinate point onto the normalized spherical surface to obtain a third projection coordinate corresponding to the first three-dimensional coordinate point and a fourth projection coordinate corresponding to the second three-dimensional coordinate point, respectively;

[0010] calculating the pose of the two-dimensional camera and the pose of the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate, and the fourth projection coordinate.

[0011] Embodiments of the second aspect of the present application provide a pose calculation system of a camera, comprising:

[0012] a visible light camera configured to capture a visible light image of a real scene;

[0013] a depth camera configured to capture a depth image of the real scene;

[0014] a processor configured to obtain a first image coordinate point corresponding to a first three-dimensional coordinate point in the real scene from a visible light image captured by the visible light camera, and obtain a second image coordinate point corresponding to a second three-dimensional coordinate point in the real scene from a depth image captured by the depth camera, project the first image coordinate point and the second image coordinate point onto a normalized spherical surface to obtain a first projection coordinate corresponding to the first image coordinate point and a second projection coordinate corresponding to the second image coordinate point, respectively, project the first three-dimensional coordinate point and the second three-dimensional coordinate point onto the normalized spherical surface to obtain a third projection coordinate corresponding to the first three-dimensional coordinate point and a fourth projection coordinate corresponding to the second three-dimensional coordinate point, and calculate the pose of the two-dimensional camera and the pose of the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate, and the fourth projection coordinate.

[0015] The center of the normalized spherical surface is an optical center of the visible light camera.

[0016] Embodiments of the third aspect of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect.

[0017] Embodiments of the fourth aspect of the present application provide a computer readable storage medium, which stores a computer program executable by a processor to implement the method of the first aspect.

[0018] The embodiment of the fifth aspect of the present application provides a computer program product comprising a computer program, which is executed by a processor to implement the method of the first aspect.

[0019] The technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0020] In the embodiments of the present application, the image coordinate points captured by the two-dimensional camera, the image coordinate points captured by the depth camera and the three-dimensional coordinate points in the real scene being captured are all projected onto the same standardized spherical surface, the pose calculation of the two-dimensional camera and the depth camera is unified into the same calculation process, and the poses of the two-dimensional camera and the depth camera can be calculated synchronously, with small calculation amount and high calculation efficiency. Projecting the image coordinate points and the three-dimensional coordinate points onto the standardized spherical surface makes the contribution of all points to the pose the same, reduces the adverse effects of image distortion caused by the camera on the pose calculation of the camera, makes the optimization iteration of the pose more easily convergent, makes the calculation process more reasonable, reduces the possibility of the algorithm falling into a local minimum value when iteratively calculating the pose of the camera, and improves the accuracy of the pose calculation. Moreover, the image captured by the depth camera has depth information, can make the size of the constructed map have a corresponding relationship with the real size of the three-dimensional space, and improves the working precision of the robot in tracking or positioning.

[0021] Additional aspects and advantages will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the application. Furthermore, the same reference numerals are used throughout the several views of the drawings to designate the same or similar parts. In the drawings:

[0023] Figure 1 A geometric view of projection of three-dimensional coordinate points in a real scene to a camera image is shown;

[0024] Figure 2 A schematic view showing that the two-dimensional camera and the depth camera on the robot provided by the embodiments of the present application do not have a common view relationship is shown;

[0025] Figure 3 A schematic view showing that the two-dimensional camera and the depth camera on the robot provided by the embodiments of the present application have a field of view overlap area is shown;

[0026] Figure 4A flow chart of a camera pose calculation method based on multi-class image processing is shown.

[0027] Figure 5 A schematic diagram of projecting an image coordinate point on an image captured by a camera onto a standardized spherical surface is shown.

[0028] Figure 6 A structural schematic diagram of a camera pose calculation device based on multi-class image processing is shown.

[0029] Figure 7 A structural schematic diagram of a camera pose calculation system is shown.

[0030] Figure 8 A structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and so that the scope of the present application can be accurately conveyed to those skilled in the art.

[0032] First, the terms used in the embodiments of the present application are explained:

[0033] Image pixel coordinate system: a planar coordinate system constructed in an image captured by a camera, with the upper left corner point as the image coordinate origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis. An image pixel coordinate system can also be constructed with other corner points in the image as the coordinate origin.

[0034] Camera coordinate system: a coordinate system with the camera optical center as the coordinate origin, the front of the camera's field of view as the z-axis, and the x and y axes respectively the same as the x and y axes of the image coordinate system.

[0035] World coordinate system: a fixed coordinate system in three-dimensional space, used as a reference coordinate system in three-dimensional space to describe the position of objects in three-dimensional space.

[0036] Pre-set conversion matrix: a conversion matrix between the camera coordinate system of a two-dimensional camera and the camera coordinate system of a depth camera.

[0037] Standardized spherical surface: a spherical surface with a two-dimensional camera optical center as the spherical center and a unit length as the radius, which is a two-dimensional manifold embedded in the preset three-dimensional coordinate system. The unit length can be 1 meter or 1.5 meters, etc.

[0038] The first three-dimensional coordinate point is a three-dimensional coordinate point in a real scene photographed by a two-dimensional camera.

[0039] The first image coordinate point is an image coordinate point corresponding to the first three-dimensional coordinate point in a two-dimensional image photographed by the two-dimensional camera.

[0040] The first projection coordinate is a projection coordinate of the first image coordinate point in the two-dimensional image photographed by the two-dimensional camera on a normalized spherical surface.

[0041] The third projection coordinate is a projection coordinate of the first three-dimensional coordinate point on the normalized spherical surface.

[0042] The second three-dimensional coordinate point is a three-dimensional coordinate point in a real scene photographed by a depth camera.

[0043] The second image coordinate point is an image coordinate point corresponding to the second three-dimensional coordinate point in a depth image photographed by the depth camera.

[0044] The second projection coordinate is a projection coordinate of the second image coordinate point in the depth image photographed by the depth camera on the normalized spherical surface.

[0045] The fourth projection coordinate is a projection coordinate of the second three-dimensional coordinate point on the normalized spherical surface.

[0046] In the related art, in the process of calculating the pose of a single camera, a three-dimensional coordinate point and an image coordinate point corresponding to the three-dimensional coordinate point are projected onto a normalized plane of the camera, to obtain a projection coordinate of the three-dimensional coordinate point and a projection coordinate of the image coordinate point corresponding to the three-dimensional coordinate point. The pose of the camera is solved based on the re-projection error of the projection of the two onto the normalized plane. Figure 1 A geometric view of the projection of a three-dimensional coordinate point in a real scene to a camera image is shown. An intersection between a line connecting the three-dimensional coordinate point and a camera optical center and an imaging plane of the camera is an image coordinate point corresponding to the three-dimensional coordinate point in an image photographed by the camera.

[0047] Many robots are provided with multiple cameras. The related art calculates the poses of the multiple cameras respectively by using the above method. The calculation processes of the cameras are independent of and do not affect each other, resulting in a large amount of calculation and low calculation efficiency of the poses of the multiple cameras on the same robot.

[0048] Based on this, the embodiment of the present application provides a camera pose calculation method, system and device based on multi-type image processing. In order to facilitate understanding of the camera pose estimation process provided by the embodiment of the present application, the application scenario of the embodiment of the present application is first introduced.

[0049] In this embodiment, multiple cameras are installed on some electronic devices, including robots, automobiles, mobile phones, visual monitoring devices, etc. The multiple cameras installed on the electronic devices include two-dimensional cameras and depth cameras. The two-dimensional cameras can include visible light cameras, such as pinhole cameras or fisheye cameras. The images captured by the two-dimensional cameras are two-dimensional images, and the images captured by the depth cameras are depth images. The depth image includes a standard RGB three-channel color image and a depth map. The pixel value of each pixel in the depth map represents the actual distance between the corresponding three-dimensional coordinate point in the real scene and the depth camera.

[0050] A 2D camera and a depth camera mounted on an electronic device may not have a shared field of view; that is, their fields of view do not overlap. Figure 2 As shown in the diagram, the area between the two dashed lines extending from the depth camera represents the field of view of the depth camera, and the area between the two dashed lines extending from the 2D camera represents the field of view of the 2D camera. Alternatively, the fields of view of the 2D camera and the depth camera may have a small overlap, such as... Figure 3 As shown, the shaded area represents the overlapping field of view of the 2D camera and the depth camera. The area of ​​the overlapping area can be smaller than a preset percentage of the field of view of the 2D camera, or the area of ​​the overlapping area can be smaller than a preset percentage of the field of view of the depth camera. The preset percentage can be 1%, 3%, or 5%, etc.

[0051] When the 2D camera and the depth camera do not share a common field of view, the 2D image captured by the 2D camera and the depth image captured by the depth camera will not contain pixels corresponding to the same 3D coordinate point in the real-world scene. When the fields of view of the 2D camera and the depth camera overlap, during the calculation of the camera pose using the method provided in this application, the 3D coordinate points in the overlapping area can be calculated using the image coordinate points corresponding to these 3D coordinate points in the 2D image captured by the 2D camera. Alternatively, the image coordinate points corresponding to these 3D coordinate points in the depth image captured by the depth camera can also be used for calculation. The key is to ensure that the image coordinate points involved in the calculation in the 2D image and the image coordinate points involved in the calculation in the depth image do not correspond to the same 3D coordinate points in the real-world scene.

[0052] In the application scenarios applicable to the embodiments of this application, electronic devices equipped with 2D cameras and depth cameras can be in a stationary state. In the stationary state, the 2D cameras and depth cameras capture images of the real scene. The embodiments of this application calculate the pose of the 2D cameras and depth cameras based on the captured images.

[0053] In another application scenario, the two-dimensional camera and the depth camera can also be in a motion state, for example, the electronic device provided with the two-dimensional camera and the depth camera is movable, or the electronic device is in a static state but the two-dimensional camera and the depth camera are movable relative to the electronic device. The two-dimensional camera and the depth camera can capture a plurality of images at a plurality of moments in the motion process, and embodiments of the present application calculate the poses of the two-dimensional camera and the depth camera at each moment based on the images captured at each moment.

[0054] Embodiments of the present application project the image coordinate points and the corresponding three-dimensional coordinate points of the two-dimensional camera and the depth camera onto the same standardized spherical surface, and unify the pose calculation processes of the two-dimensional camera and the depth camera into the same calculation process. Compared with the manner in which the pose calculation of the two-dimensional camera and the pose calculation of the depth camera are performed independently, the calculation amount is greatly reduced, and the pose calculation efficiency is improved. Moreover, the distance between each point on the standardized spherical surface and the center of the sphere is equal, which can eliminate the influence of different degrees of distortion caused by image coordinate points at different positions, so that the influence of each image coordinate point on the pose calculation of the camera is the same, thereby improving the accuracy of the pose calculation. Moreover, the image captured by the depth camera has depth information, which can make the size of the constructed map have a counterpart relationship with the real size of the three-dimensional space, and improve the working accuracy of the robot in tracking or positioning.

[0055] The scheme of the embodiments of the present application will be described in detail below with reference to the drawings.

[0056] Referring to Figure 4 A flowchart of a camera pose calculation method based on multi-type image processing is shown, and the method specifically includes the following steps:

[0057] Step 101: obtaining a first image coordinate point corresponding to a first three-dimensional coordinate point in a real scene from a two-dimensional image captured by a two-dimensional camera; and obtaining a second image coordinate point corresponding to a second three-dimensional coordinate point in the real scene from a depth image captured by a depth camera.

[0058] The above real scene can be any scene in a three-dimensional space, and a plurality of three-dimensional coordinate points in the world coordinate system can be calculated in the real scene by triangulation or artificial measurement. The two-dimensional image and the depth image obtained by capturing the real scene by the two-dimensional camera and the depth camera contain the image coordinate points corresponding to the three-dimensional coordinate points calculated in advance. Since there is no common view relationship between the two-dimensional camera and the depth camera, or the overlapping area of the view fields of the two-dimensional camera and the depth camera is very small, the same three-dimensional coordinate points captured by the two-dimensional camera and the depth camera are very few. Embodiments of the present application assume that the same three-dimensional coordinate points captured by the two-dimensional camera and the depth camera are captured only by the two-dimensional camera or only by the depth camera.

[0059] In some embodiments, the correspondence between the image coordinate points in the image and the three-dimensional coordinate points in the three-dimensional space can be determined in the following manner. Specifically, a calibration board such as a checkerboard is given in the image, and a world coordinate system is determined based on the calibration board, for example, with the first corner point of the checkerboard as the origin of the world coordinate system, the long side of the checkerboard as the x-axis, and the short side as the y-axis. If the long side and the short side of the checkerboard are equal, any one side can be selected as the x-axis. The direction of the z-axis is determined according to the selected x-axis, y-axis, and right-hand screw rule. In the established coordinate system, the spacing of the checkerboard is determined, so the three-dimensional coordinates of each corner point of the checkerboard in the coordinate system are determined, thereby obtaining the three-dimensional coordinates of each corner point of the checkerboard. In the image, the image coordinates of each corner point of the checkerboard can also be determined based on the image pixel coordinate system, so that the three-dimensional coordinates of the same corner point in the world coordinate system and the image coordinates on the image are obtained, thereby obtaining the correspondence between the multiple sets of image coordinate points and the three-dimensional coordinate points.

[0060] In other embodiments, the same point in the three-dimensional space is captured by the camera at two different times, and the relative positional relationship between the cameras at the two different times is known. In this case, the three-dimensional coordinate point in the three-dimensional space is calculated using the binocular imaging principle triangulation. For the images captured by the cameras at the two different times, feature points are extracted from the images, and a unique descriptor is extracted for each feature point. If the similarity between the descriptor of a point in one image and the descriptor of a point in another image is greater than a threshold, it is determined that the two points in the two images are captured from the same point in the three-dimensional space. Thus, a one-to-one mapping relationship between the image coordinate points in the image and the three-dimensional coordinate points in the three-dimensional space can be determined.

[0061] In the embodiments of the present application, the first image coordinate point corresponding to the first three-dimensional coordinate point in the two-dimensional image and the second image coordinate point corresponding to the second three-dimensional coordinate point in the depth image can be determined in any of the above manners. Each time of pose calculation requires at least three sets of point pairs corresponding to the two-dimensional image and three sets of point pairs corresponding to the depth image.

[0062] Step 102: Projecting the first image coordinate point and the second image coordinate point onto a standardized spherical surface to obtain a first projection coordinate corresponding to the first image coordinate point and a second projection coordinate corresponding to the second image coordinate point, respectively, the center of the standardized spherical surface being the optical center of the two-dimensional camera.

[0063] Since the second image coordinate point in the depth image shot by the depth camera has depth, projecting the second image coordinate point to the imaging plane of the two-dimensional camera can obtain a unique corresponding projection point. However, the first image coordinate point in the two-dimensional image shot by the two-dimensional camera does not have depth, and if the first image coordinate point is projected to the imaging plane of the depth camera, the corresponding projection point cannot be determined because the first image coordinate point does not have depth. Based on this, an embodiment of the present application constructs a standardized spherical surface corresponding to the two-dimensional camera with the optical center of the two-dimensional camera as the spherical center and with a unit length as the radius. The first image coordinate point in the two-dimensional image and the second image coordinate point in the depth image are both projected to the standardized spherical surface.

[0064] Figure 5 A schematic diagram of projecting an image coordinate point on an image shot by a camera to a standardized spherical surface is shown. As shown in FIG. 1, a three-dimensional coordinate point in a world coordinate system has a unique intersection point with the standardized spherical surface after being projected to the image of the camera, and the intersection point is the projection coordinate of the corresponding image coordinate point on the standardized spherical surface. Figure 5

[0065] After obtaining the plurality of first image coordinate points in the two-dimensional image and the plurality of second image coordinate points in the depth image through step 101, each obtained first image coordinate point is projected to the standardized spherical surface to obtain a first projection coordinate corresponding to each first image coordinate point. Each obtained second image coordinate point is also projected to the standardized spherical surface to obtain a second projection coordinate corresponding to each second image coordinate point.

[0066] The projection process of the first image coordinate point in the two-dimensional image is different from the projection process of the second image coordinate point in the depth image, and the two projection processes are described below.

[0067] For the first image coordinate point in the two-dimensional image, the first image coordinate point is first projected to a standardized plane to obtain a coordinate point corresponding to the first image coordinate point on the standardized plane. The standardized plane is a tangent plane of the standardized spherical surface described above. Then, a transformation parameter for converting the standardized plane coordinate to the standardized spherical surface coordinate is calculated. Based on the calculated transformation parameter, the coordinate point on the standardized plane corresponding to the first image coordinate point is converted to the standardized spherical surface to obtain the first projection coordinate of the first image coordinate point on the standardized spherical surface.

[0068] The specific process of projecting the first image coordinate point to the standardized spherical surface is described below by taking a pinhole camera as an example. In this specific example, the optical center of the pinhole camera is taken as the spherical center to construct the standardized spherical surface, and a tangent plane of the standardized spherical surface is taken as the standardized plane. The tangent plane can be a tangent plane parallel to the imaging plane of the pinhole camera. ​

[0069] The first step is to project the first image coordinate point in the two-dimensional image captured by the pinhole camera onto the standardized plane using the following formula (1).

[0070]

[0071] In formula (1), p x and p y These are the x and y coordinates of the first image coordinate point, respectively. x f y c x and c y This is the camera's internal reference data. The coordinates of the first image coordinate point (x, y) projected onto the normalized plane.

[0072] The second step is to calculate the transformation parameters from standardized planar coordinates to standardized spherical coordinates using the following formula (2).

[0073]

[0074] In formula (2), β=θ-α,

[0075] Where s is the transformation parameter mentioned above. for The first two coordinates and The sum of squares. k0, k1, k2, and k3 are the distortion coefficients of the pinhole camera.

[0076] The third step is to transform the coordinate points on the normalized plane corresponding to the first image coordinate points to the normalized sphere based on the calculated transformation parameters using the following formula (3).

[0077]

[0078] In formula (3), i is the number of the three-dimensional coordinate point in the real scene, 1≤i≤K. Assume the number of the first three-dimensional coordinate points captured by the pinhole camera is M. f The number of second 3D coordinate points captured by the depth camera is N. r Then N f +N r =K. Assuming the three-dimensional coordinate points are numbered sequentially starting from 1, and the number of the first three-dimensional coordinate point is greater than the number of the second three-dimensional coordinate point, then N... r When +1≤i≤K, p i The first projected coordinates are the coordinates of the first image point projected onto the normalized sphere.

[0079] For each first image coordinate point obtained in step 101, each first image coordinate point is projected onto the normalized sphere respectively by the above manner, and a plurality of corresponding relationships between first image coordinate points and first projection coordinates are obtained.

[0080] For the process of projecting the second image coordinate point in the depth image onto the normalized sphere, since the second image coordinate point in the depth image includes a pixel coordinate and a depth, the second image coordinate point is equivalent to a three-dimensional coordinate. The specific process of projecting the second image coordinate point onto the normalized sphere is that, based on the camera internal parameter of the depth camera and the pixel coordinate and the depth included in the second image coordinate point, a planar projection coordinate of the pixel coordinate on the normalized plane is calculated. Based on the depth included in the second image coordinate point, the calculated planar projection coordinate, and the preset conversion matrix, a pixel projection coordinate of the pixel coordinate projected onto the normalized sphere and a projection depth of the depth projected onto the normalized sphere are calculated. The pixel projection coordinate and the projection depth are determined as the second projection coordinate corresponding to the second image coordinate point.

[0081] Specifically, the second projection coordinate corresponding to the second image coordinate point can be calculated by the following formula (4).

[0082]

[0083] In the formula (4), 0≤i≤N r , p i is the pixel projection coordinate of the pixel coordinate of the i th second image coordinate point projected onto the normalized sphere. is the planar projection coordinate of the pixel coordinate of the i th second image coordinate point on the normalized plane, p rxi , p ryi represents the pixel coordinate of the i th second image coordinate point. f rx , f ry , c rx and c ry are the camera internal parameters of the depth camera. is the depth of the i th second image coordinate point, l di is the projection depth of the depth of the i th second image coordinate point on the normalized sphere. The pixel projection coordinate p i and the projection depth l di constitute the second projection coordinate corresponding to the second image coordinate point. T fd is the preset conversion matrix between the camera coordinate system of the pinhole camera and the camera coordinate system of the depth camera.

[0084] For each second image coordinate point obtained in step 101, each second image coordinate point is projected onto the normalized sphere respectively by the above manner, and a plurality of corresponding relationships between second image coordinate points and second projection coordinates are obtained.

[0085] Step 103: Projecting the first three-dimensional coordinate point and the second three-dimensional coordinate point onto the normalized sphere to obtain a third projection coordinate corresponding to the first three-dimensional coordinate point and a fourth projection coordinate corresponding to the second three-dimensional coordinate point respectively.

[0086] Firstly, based on the first projection coordinate corresponding to the first image coordinate point and the first three-dimensional coordinate point, an initial value of the pose of the two-dimensional camera is calculated.

[0087] Specifically, based on the first projection coordinate corresponding to each first image coordinate point and the first three-dimensional coordinate point corresponding to each first image coordinate point, an initial value of a rotation matrix between the world coordinate system and the camera coordinate system of the two-dimensional camera is calculated. An initial value of a displacement vector between the world coordinate system and the camera coordinate system of the two-dimensional camera is set to zero. Based on the initial value of the rotation matrix and the initial value of the displacement vector, a conversion matrix between the world coordinate system and the camera coordinate system of the two-dimensional camera is constructed, and the conversion matrix is the initial value of the pose of the two-dimensional camera.

[0088] The first projection coordinate corresponding to each first image coordinate point and the first three-dimensional coordinate point corresponding to each first image coordinate point can be input into a preset pose estimation algorithm to obtain the initial value of the rotation matrix. The preset pose estimation algorithm can include a PnP (Perspective-n-Point, multi-point perspective imaging) algorithm, such as a P3P algorithm.

[0089] The initial value of the pose constructed by the above manner can be represented by the following formula (5).

[0090]

[0091] In formula (5), T cw is the initial value of the pose of the two-dimensional camera, R cw is the initial value of the rotation matrix, and t cw is the initial value of the displacement vector.

[0092] After obtaining the initial value of the pose of the two-dimensional camera, based on the initial value of the pose of the two-dimensional camera and the first three-dimensional coordinate point, the third projection coordinate obtained by projecting the first three-dimensional coordinate point onto the normalized sphere is calculated. Based on the initial value of the pose of the two-dimensional camera and the second three-dimensional coordinate point, the fourth projection coordinate obtained by projecting the second three-dimensional coordinate point onto the normalized sphere is calculated.

[0093] The calculation manners of each three-dimensional coordinate point projected to the normalized sphere are same for the first three-dimensional coordinate point and the second three-dimensional coordinate point, and thus the specific calculation process is described by taking one three-dimensional coordinate point as an example. The application takes any one first three-dimensional coordinate point as an example for description.

[0094] Specifically, the product of the initial value of the pose of the two-dimensional camera and the coordinate vector of the first three-dimensional coordinate point is calculated to obtain a target coordinate point corresponding to the first three-dimensional coordinate point in the camera coordinate system of the two-dimensional camera. The norm of the coordinate vector of the target coordinate point is taken, and the ratio between the coordinate vector of the target coordinate point and the norm is calculated to obtain the third projection coordinate of the first three-dimensional coordinate point on the normalized sphere.

[0095] The projection coordinate of the three-dimensional coordinate point can be calculated by the projection function of the three-dimensional coordinate point shown in the following formula (6).

[0096]

[0097] In the formula (6), P wi is the i-th three-dimensional coordinate point, p ci is the projection coordinate of the i-th three-dimensional coordinate point projected on the normalized sphere, T cw is the initial value of the pose of the two-dimensional camera. When 1≤i≤N r , p ci is the fourth projection coordinate corresponding to the i-th second three-dimensional coordinate point. When N r +1≤i≤K, p ci is the third projection coordinate corresponding to the i-th first three-dimensional coordinate point.

[0098] For each first three-dimensional coordinate point and second three-dimensional coordinate point obtained in step 101, the third projection coordinate corresponding to each first three-dimensional coordinate point is calculated and the fourth projection coordinate corresponding to each second three-dimensional coordinate point is calculated by the above-mentioned manner.

[0099] In the above calculation process, the complete coordinate information of the three-dimensional coordinate point is used, which can make the calculation of the projection coordinate of the three-dimensional coordinate point in the world coordinate system on the normalized sphere more accurate, and thus can reduce the situation that the pose optimization iteration falls into a local minimum value when the calculated projection coordinate is used for subsequent camera pose calculation, and improve the accuracy of the pose calculation.

[0100] The embodiments of the present application do not limit the execution order of steps 102 and 103, and the two steps can be executed synchronously or in any order in actual application. The related points of the two-dimensional camera and the related points of the depth camera are projected onto the standardized spherical surface corresponding to the two-dimensional camera through steps 102 and 103, the pose calculation of the two-dimensional camera and the pose calculation of the depth camera are unified into the same calculation process, the calculation amount of the pose calculation of the multiple cameras arranged on the electronic device is simplified, and the efficiency of the multi-camera pose calculation is improved.

[0101] Step 104: calculating the pose of the two-dimensional camera and the pose of the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate.

[0102] First, the first total projection error corresponding to the two-dimensional camera and the depth camera is calculated based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate.

[0103] Specifically, the difference between the first projection coordinate corresponding to the first image coordinate point and the third projection coordinate of the first three-dimensional coordinate point corresponding to the first image coordinate point is calculated to obtain the projection error corresponding to the first three-dimensional coordinate point. And the difference between the second projection coordinate corresponding to the second image coordinate point and the fourth projection coordinate of the second three-dimensional coordinate point corresponding to the second image coordinate point is calculated to obtain the projection error corresponding to the second three-dimensional coordinate point.

[0104] For the projection error of the first three-dimensional coordinate point photographed by the two-dimensional camera, the projection error of the first three-dimensional coordinate point can be calculated by the following formula (7).

[0105] e i =p i -p ci ,N r +1≤i≤K…(7)

[0106] In formula (7), e i is the projection error corresponding to the i th first three-dimensional coordinate point, p i is the first projection coordinate of the i th first three-dimensional coordinate point corresponding to the first image coordinate point on the standardized spherical surface, p ci is the third projection coordinate of the i th first three-dimensional coordinate point on the standardized spherical surface.

[0107] For the projection error of the second three-dimensional coordinate point photographed by the depth camera, the projection error of the second three-dimensional coordinate point can be calculated by the following formula (8).

[0108]

[0109] In formula (8), e i is the projection error corresponding to the i-th second three-dimensional coordinate point, p i is the second projection coordinate of the i-th second three-dimensional coordinate point on the normalized spherical surface, p i (x) is the x coordinate of the second projection coordinate p i , p ci is the fourth projection coordinate of the i-th second three-dimensional coordinate point on the normalized spherical surface, p ci (x) is the x coordinate of the fourth projection coordinate p ci , and bf is a constant.

[0110] The projection error corresponding to each first three-dimensional coordinate point and the projection error corresponding to each second three-dimensional coordinate point are calculated respectively by the above method, and then the first total projection error corresponding to the two-dimensional camera and the depth camera is calculated based on the projection error corresponding to each first three-dimensional coordinate point and the projection error corresponding to each second three-dimensional coordinate point.

[0111] The first total projection error can be obtained by summing the projection error corresponding to each first three-dimensional coordinate point and the projection error corresponding to each second three-dimensional coordinate point. Alternatively, the first total projection error can also be calculated by formula (9) as follows.

[0112]

[0113] In formula (9), error1 is the first total projection error, e i is the projection error corresponding to the i-th three-dimensional coordinate point.

[0114] The first total projection error is optimized to obtain the target pose value of the two-dimensional camera when the first total projection error takes the minimum value, and the target pose value is the finally calculated pose of the two-dimensional camera. The pose of the depth camera is calculated based on the preset conversion matrix and the target pose value of the two-dimensional camera.

[0115] Before optimizing the first total projection error, the Jacobian matrix of the projection error with respect to the optimization variable is also calculated, where the optimization variable is the pose of the two-dimensional camera.

[0116] Specifically, based on the initial pose value of the 2D camera, the first 3D coordinate point, and the corresponding projection error of the first 3D coordinate point, a first Jacobian matrix of the projection error of the first 3D coordinate point relative to the initial pose value of the 2D camera is calculated. Similarly, based on the initial pose value of the 2D camera, the second 3D coordinate point, and the corresponding projection error of the second 3D coordinate point, a first Jacobian matrix of the projection error of the second 3D coordinate point relative to the initial pose value of the 2D camera is calculated. Then, based on the initial pose value of the 2D camera, the first total projection error, the first Jacobian matrix corresponding to the first 3D coordinate point, and the first Jacobian matrix corresponding to the second 3D coordinate point, the first total projection error is optimized using a preset optimization algorithm.

[0117] The calculation process for the first Jacobian matrix corresponding to the first 3D coordinate point captured by the 2D camera is as follows:

[0118] i is the number of the three-dimensional coordinate point, when N r When +1≤i≤K, i is the number of the first 3D coordinate point captured by the 2D camera. Let: P ci =T cw P wi =[X ci ,Y ci Z ci ] T P ci Let be the third projected coordinate of the i-th first three-dimensional coordinate point on the normalized sphere. The projection error e corresponding to the i-th first three-dimensional coordinate point is calculated using the following formula (10). i For the third projected coordinate P ci The first Lie algebra derivative is obtained. And the third projected coordinate P is calculated using formula (11). ci The second Lie algebra is derived with respect to the initial pose of the 2D camera. The product of the derivatives of the first and second Lie algebras is calculated using formula (12) to obtain the projection error e corresponding to the i-th first 3D coordinate point. i The first Jacobian matrix for the initial pose of the 2D camera.

[0119]

[0120]

[0121]

[0122] In formulas (10), (11), and (12), I is the identity matrix. X ci ,ci and Zci are the third projected coordinates P ci The x, y, and z coordinates.

[0123] The calculation process for the first Jacobian matrix corresponding to the second 3D coordinate point captured by the depth camera is as follows: when 1≤i≤N r Let i be the number of the second 3D coordinate point captured by the depth camera, and P be the number of the second 3D coordinate point captured by the depth camera. ci Let be the fourth projected coordinate of the i-th second three-dimensional coordinate point on the normalized sphere. The projection error e corresponding to the i-th second three-dimensional coordinate point is calculated using the following formula (13). i For the fourth projected coordinate P ci The first Lie algebra derivative is obtained. And the fourth projected coordinate P is calculated using formula (14). ci The second Lie algebra is derived with respect to the initial pose of the 2D camera. The product of the derivatives of the first and second Lie algebras is calculated using formula (15) to obtain the projection error e corresponding to the i-th second 3D coordinate point. i The first Jacobian matrix for the initial pose of the 2D camera.

[0124]

[0125]

[0126]

[0127] In formulas (13), (14) and (15), I is the identity matrix, X ci ,ci and Z ci The fourth projection coordinates P are respectively ci The x, y, and z coordinates are given, and bf is a constant.

[0128] The first Jacobian matrix corresponding to each first three-dimensional coordinate point captured by the two-dimensional camera and the first Jacobian matrix corresponding to each second three-dimensional coordinate point captured by the depth camera are calculated using the above method.

[0129] The steps of calculating the first total projection error and the first Jacobian matrix can be performed simultaneously or sequentially in any order.

[0130] After obtaining the first total projection error and all the first Jacobian matrices required for optimization, based on the initial pose value of the 2D camera, the first total projection error, the first Jacobian matrix corresponding to the first 3D coordinate point, and the first Jacobian matrix corresponding to the second 3D coordinate point, the first total projection error is optimized iteratively using a preset optimization algorithm to obtain the target pose value of the 2D camera when the first total projection error reaches its minimum value. The preset optimization algorithm can be the LM (Levenberg Marquarelt) algorithm, Newton's method, or Gauss-Newton's method, etc. The optimization model implemented using the preset optimization algorithm is shown in formula (16).

[0131]

[0132] In formula (16), SE3 represents all rigid transformation spaces, and error1 is the first total projection error.

[0133] The minimum value of the first total projection error is solved in the above manner, and a pose value of the two-dimensional camera when the first total projection error reaches the minimum value is obtained. The pose value is the calculated pose of the two-dimensional camera, which is referred to as a target pose value of the two-dimensional camera in the embodiment of the application.

[0134] Since the pose of the two-dimensional camera is a conversion matrix between the world coordinate system and the camera coordinate system of the two-dimensional camera, and the preset conversion matrix is a conversion matrix between the camera coordinate system of the two-dimensional camera and the camera coordinate system of the depth camera, the conversion matrix between the world coordinate system and the camera coordinate system of the depth camera can be calculated based on the preset conversion matrix and the target pose value of the two-dimensional camera. The conversion matrix is the pose of the depth camera.

[0135] In the embodiment of the application, one two-dimensional camera and at least one depth camera can be installed on an electronic device. The pose of the one two-dimensional camera and the pose of each depth camera can be calculated simultaneously by the method of the embodiment of the application, without calculating the pose of each camera one by one, thereby improving the pose calculation efficiency in a multi-camera scene.

[0136] The image coordinate points captured by the two-dimensional camera, the image coordinate points captured by the depth camera, and the three-dimensional coordinate points in the real scene are projected onto the same standardized spherical surface in the embodiment of the application. The pose calculation of the two-dimensional camera and the depth camera is unified into the same calculation process, and the poses of the two-dimensional camera and the depth camera can be calculated simultaneously. The calculation amount is small, and the calculation efficiency is high. By projecting onto the standardized spherical surface, the adverse effects of image distortion caused by camera shooting on camera pose calculation are reduced, the optimization iteration of the pose is more easily converged, the calculation process is more reasonable, the algorithm is less likely to fall into a local minimum value during the iterative calculation of the camera pose, and the accuracy of the pose calculation is improved. Moreover, the image captured by the depth camera has depth information, which can make the size of the constructed map have a corresponding relationship with the real size of the three-dimensional space, and improve the working accuracy of the robot in tracking or positioning.

[0137] By the multi-camera pose calculation method provided in the above embodiments, whenever the two-dimensional camera and the depth camera on the device capture images in a real scene, the current poses of the two-dimensional camera and the depth camera can be calculated by the method of the above embodiments. In another application scenario, considering that a robot or a vehicle or the like often moves, the cameras arranged on the device move together with the device, and the movement of the cameras can affect the pose calculation accuracy. Therefore, in some other embodiments of the present application, the poses of the two-dimensional camera and the depth camera at multiple moments during the movement are optimized and adjusted as a whole.

[0138] The two-dimensional camera and the depth camera arranged on the device capture the real scene at different moments during the movement. The two-dimensional camera captures a plurality of two-dimensional images of the real scene at the multiple moments, and the depth camera captures a plurality of depth images of the real scene at the multiple moments.

[0139] It is assumed that the two-dimensional camera and the depth camera capture images at each of the T moments during the movement, and there are T two-dimensional images and T depth images in total. For K three-dimensional coordinate points in the real scene, the same camera can capture all the K three-dimensional coordinate points or only part of the K three-dimensional coordinate points at different moments of the T moments. Wherein, T≥2.

[0140] For each of the T two-dimensional images and each of the T depth images, some processing needs to be performed on each two-dimensional image and each depth image, and the processing process of each two-dimensional image is the same, and the processing process of each depth image is also the same. Therefore, the embodiments of the present application take the two-dimensional image captured by the two-dimensional camera at the i th moment and the depth image captured by the depth camera at the i th moment as an example to illustrate the processing process.

[0141] Based on the two-dimensional image and the depth image at the i th moment, by the operations of steps 101-103 in the above embodiments, each first three-dimensional coordinate point captured by the two-dimensional camera at the i th moment, the first image coordinate corresponding to each first three-dimensional coordinate point, the first projection coordinate corresponding to each first image coordinate, and the third projection coordinate corresponding to each first three-dimensional coordinate point are obtained. In addition, each second three-dimensional coordinate point captured by the depth camera at the i th moment, the second image coordinate corresponding to each second three-dimensional coordinate point, the second projection coordinate corresponding to each second image coordinate, and the fourth projection coordinate corresponding to each second three-dimensional coordinate point are obtained. In addition, based on each first three-dimensional coordinate point and each first projection coordinate at the i th moment, the initial value of the pose of the two-dimensional camera at the i th moment is calculated.

[0142] In the embodiments of the present application, P wkrepresents the kth three-dimensional coordinate point. Since the three-dimensional coordinate points captured by the camera at each time instant can only be a part of all the three-dimensional coordinate points calculated in advance, a variable μ is added ijk represents the three-dimensional coordinate point P wk whether it is captured by the camera at the ith time instant, j represents the three-dimensional coordinate point P wk the number of the corresponding image coordinate point in the image captured by the camera. The variable μ is represented by the following formula (17) ijk .

[0143]

[0144] Suppose that the camera captures N i three-dimensional coordinate points at the ith time instant, where the depth camera captures N ri second three-dimensional coordinate points, the two-dimensional camera captures N fi first three-dimensional coordinate points, N i =N ri +N fi . All the three-dimensional coordinate points captured by the camera at the ith time instant are sequentially numbered. Then 1≤k≤N ri represents the second three-dimensional coordinate point captured by the depth camera. When N ri +1≤k≤N i , it represents the first three-dimensional coordinate point captured by the two-dimensional camera.

[0145] Let represent the kth three-dimensional coordinate point P wk the corresponding image coordinate point in the image captured by the camera at the ith time instant, j represents that the image coordinate point is the jth point among the image coordinate points extracted from the image captured by the camera at the ith time instant. Let p ijk represent the image coordinate point the fourth projection coordinate obtained by projecting on the normalized sphere. When 1≤j≤N ri , it represents the jth second image coordinate point in the depth image at the ith time instant, p ijk represents the second image coordinate point the fourth projection coordinate obtained by projecting on the normalized sphere. When N ri +1≤j≤N i , it represents the jth first image coordinate point in the two-dimensional image at the ith time instant, p ijk represents the first image coordinate point the third projection coordinate obtained by projecting on the normalized sphere. Let T iw represent the initial value of the pose of the two-dimensional camera at the ith time instant.

[0146] Let eijk representing image coordinate points projected coordinates on the normalized sphere p ijk with three-dimensional coordinate points P wk projected error between the projected coordinates on the normalized sphere. For the corresponding projected error of the depth camera, it can be calculated by formula (18).

[0147]

[0148] In formula (18), P cik = T iw P wk = [X cik , Y cik , Z ciik ] T , p ijk (x) represents the x coordinate of p ijk , p cik (x) represents the x coordinate of p cik (x), and bf is a constant.

[0149] For the corresponding projected error of the two-dimensional camera, it can be calculated by formula (19).

[0150] e ijk = p ijk - p cik , N r +1≤j≤N i , μ ijk = 1…(19)

[0151] By the above-mentioned manner, the projected error corresponding to each first three-dimensional coordinate point captured by the two-dimensional camera at each time is calculated, and the projected error corresponding to each second three-dimensional coordinate point captured by the depth camera at each time is calculated.

[0152] Then, based on the projected error corresponding to all first three-dimensional coordinate points at each time and the projected error corresponding to all second three-dimensional coordinate points, a second total projected error is calculated. The second total projected error can be obtained by summing the projected error corresponding to all first three-dimensional coordinate points at each time and the projected error corresponding to all second three-dimensional coordinate points. Alternatively, the second total projected error error2 can also be calculated by the following formula (20).

[0153]

[0154] The second total projection error is optimized to obtain a target pose value of the two-dimensional camera corresponding to each time point when the second total projection error reaches a minimum value.

[0155] Before optimizing the second total projection error, a Jacobian matrix of the projection error with respect to an optimization variable during the optimization process is also calculated, where the optimization variable is the pose of the two-dimensional camera.

[0156] Specifically, based on the initial pose value of the two-dimensional camera at the first time point, the first three-dimensional coordinate point captured by the two-dimensional camera at the first time point, and the projection error corresponding to the first three-dimensional coordinate point at the first time point, a first Jacobian matrix of the projection error of the first three-dimensional coordinate point at the first time point with respect to the initial pose value of the two-dimensional camera at the first time point is calculated. The first time point is any time point in the plurality of time points during which the two-dimensional camera moves to capture images. In addition, based on the initial pose value of the two-dimensional camera at the first time point, the second three-dimensional coordinate point captured by the depth camera at the first time point, and the projection error corresponding to the second three-dimensional coordinate point at the first time point, a first Jacobian matrix of the projection error of the second three-dimensional coordinate point at the first time point with respect to the initial pose value of the two-dimensional camera at the first time point is calculated.

[0157] The first Jacobian matrix of the projection error of the second three-dimensional coordinate point captured by the depth camera at the first time point with respect to the initial pose value of the two-dimensional camera at the first time point can be calculated by the following formula (21).

[0158]

[0159] The first Jacobian matrix of the projection error of the first three-dimensional coordinate point captured by the two-dimensional camera at the first time point with respect to the initial pose value of the two-dimensional camera at the first time point can be calculated by the following formula (22).

[0160]

[0161] In the formulas (21) and (22),

[0162] After obtaining the second total projection error and the first Jacobian matrix at each time point, the second total projection error is iteratively optimized by a preset optimization algorithm based on the initial pose value of the two-dimensional camera at each time point, the second total projection error, and the first Jacobian matrix at each time point, to obtain a target pose value of the two-dimensional camera at each time point when the second total projection error reaches a minimum value.

[0163] The preset optimization algorithm can be an LM algorithm, a Newton method, or a Gauss-Newton method. An optimization model realized by the preset optimization algorithm is shown in the following formula (23).

[0164]

[0165] Since the pose of the two-dimensional camera is a transformation matrix between the world coordinate system and the camera coordinate system of the two-dimensional camera, the preset transformation matrix is a transformation matrix between the camera coordinate system of the two-dimensional camera and the camera coordinate system of the depth camera. Therefore, for each time, based on the preset transformation matrix and the target pose value of the two-dimensional camera at the time, the transformation matrix between the world coordinate system and the camera coordinate system of the depth camera at the time can be calculated, and the transformation matrix is the pose of the depth camera at the time.

[0166] In some other embodiments of the present application, the coordinates of the three-dimensional coordinate points captured by the two-dimensional camera and the depth camera at each time can also be optimized and adjusted in the process of calculating the pose of the two-dimensional camera and the depth camera at each time.

[0167] Specifically, the projection error of each first three-dimensional coordinate point captured by the two-dimensional camera at each time is calculated with respect to the second Jacobian matrix of each first three-dimensional coordinate point captured at each time. Taking the first time as an example for illustration, the first time is any of the above-mentioned times. Specifically, based on the initial value of the pose of the two-dimensional camera at the first time, the first three-dimensional coordinate point captured by the two-dimensional camera at the first time, and the projection error corresponding to the first three-dimensional coordinate point at the first time, the second Jacobian matrix of the projection error of the first three-dimensional coordinate point at the first time with respect to the first three-dimensional coordinate point at the first time is calculated.

[0168] The initial value of the rotation matrix is obtained from the initial value of the pose of the two-dimensional camera at the first time. Based on the initial value of the pose of the two-dimensional camera at the first time and the initial value of the rotation matrix, each first three-dimensional coordinate point captured by the camera at the first time, and the projection error corresponding to each first three-dimensional coordinate point, the second Jacobian matrix of the projection error of each first three-dimensional coordinate point captured at the first time with respect to the first three-dimensional coordinate point captured at the first time is calculated. The second Jacobian matrix can be calculated by the following formula (24).

[0169]

[0170] In formula (24), R(T iw ) is the initial value of the rotation matrix of the initial value of the pose of the two-dimensional camera at the i-th time T iw .

[0171] For each first three-dimensional coordinate point captured by the two-dimensional camera at each time, the second Jacobian matrix of the projection error corresponding to each first three-dimensional coordinate point with respect to the first three-dimensional coordinate point is calculated by the above-mentioned method.

[0172] The projection error of each second three-dimensional coordinate point captured by the depth camera at each time is also calculated with respect to the second three-dimensional coordinate point captured at each time. The second Jacobian matrix of the projection error of the second three-dimensional coordinate point captured at the first time with respect to the second three-dimensional coordinate point captured at the first time is calculated based on the initial value of the pose of the two-dimensional camera at the first time, the second three-dimensional coordinate point captured by the depth camera at the first time, and the projection error corresponding to the second three-dimensional coordinate point at the first time. Specifically, the second Jacobian matrix corresponding to the second three-dimensional coordinate point can be calculated by the following formula (25).

[0173]

[0174] For each second three-dimensional coordinate point captured by the depth camera at each time, the projection error corresponding to each second three-dimensional coordinate point is calculated with respect to the second three-dimensional coordinate point by the above-mentioned manner.

[0175] Based on the initial value of the pose of the two-dimensional camera at each time, the second total projection error, and the first Jacobian matrix and the second Jacobian matrix at each time, the second total projection error is optimized by a preset optimization algorithm. The minimization problem of the second total projection error is solved, and the specific solving formula can be as shown in formula (26).

[0176]

[0177] In formula (26), P wk ∈R 3 represents the kth three-dimensional coordinate point P wk is a coordinate point in the world coordinate system R 3 .

[0178] Based on the coordinate values of each first three-dimensional coordinate point and each second three-dimensional coordinate point at each time when the second total projection error reaches the minimum value, the coordinates of the corresponding first three-dimensional coordinate points and second three-dimensional coordinate points on the map constructed during the movement of the two-dimensional camera and the depth camera are updated. In this way, the pose optimization of the two-dimensional camera and the depth camera is realized, and the coordinates of the three-dimensional coordinate points on the constructed map are also optimized, thereby improving the accuracy of the constructed map.

[0179] In the embodiment of the present application, the poses of the two-dimensional camera and the depth camera at multiple time points in the motion process are simultaneously optimized through one optimization iteration operation, the calculation amount is reduced, and the efficiency of the pose optimization calculation is improved. The image coordinate points in the images captured by different cameras at each time point and the corresponding three-dimensional coordinate points of the image coordinate points in the world coordinate system are projected onto the same standardized spherical surface, the adverse effects of the image distortion caused by the camera image capture on the camera pose calculation are reduced, the optimization iteration of the pose is more easily converged, the calculation process is more reasonable, the algorithm can be prevented from falling into a local minimum value condition during the iterative calculation of the camera pose, the accuracy of the calculated pose is improved. Moreover, the image captured by the depth camera has depth information, the size of the constructed map has a corresponding relationship with the real size of the three-dimensional space, and the working precision of the robot in tracking or positioning is improved. The coordinates of the points in the map constructed during the motion of the camera are also optimized at the same time as the pose optimization, and the accuracy of the constructed map is improved.

[0180] Referring to Figure 6 The embodiment of the present application also provides a camera pose calculation device based on multi-type image processing, which is used for executing the camera pose calculation method based on multi-type image processing described in the above embodiment, and the device comprises:

[0181] An acquisition module 201 is configured to acquire a first image coordinate point corresponding to a first three-dimensional coordinate point in a real scene from a two-dimensional image captured by a two-dimensional camera, and acquire a second image coordinate point corresponding to a second three-dimensional coordinate point in the real scene from a depth image captured by a depth camera.

[0182] A projection module 202 is configured to project the first image coordinate point and the second image coordinate point onto a standardized spherical surface to obtain a first projection coordinate corresponding to the first image coordinate point and a second projection coordinate corresponding to the second image coordinate point, respectively; and project the first three-dimensional coordinate point and the second three-dimensional coordinate point onto the standardized spherical surface to obtain a third projection coordinate corresponding to the first three-dimensional coordinate point and a fourth projection coordinate corresponding to the second three-dimensional coordinate point, respectively; the center of the standardized spherical surface is the optical center of the two-dimensional camera.

[0183] A pose calculation module 203 is configured to calculate the pose of the two-dimensional camera and the pose of the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate.

[0184] The projection module 202 is specifically configured to calculate a plane projection coordinate of the pixel coordinate on a standardized plane based on the camera internal parameter of the depth camera and the pixel coordinate and the depth included in the second image coordinate point; the standardized plane is a tangent plane of a standardized sphere; based on the depth, the plane projection coordinate and a preset conversion matrix, the pixel projection coordinate of the pixel coordinate projected on the standardized sphere and the projection depth of the depth projected on the standardized sphere are calculated; the preset conversion matrix is a conversion matrix between a camera coordinate system of the two-dimensional camera and a camera coordinate system of the depth camera; the pixel projection coordinate and the projection depth are determined as the second projection coordinate corresponding to the second image coordinate point.

[0185] The projection module 202 is specifically configured to calculate an initial value of the pose of the two-dimensional camera based on the first projection coordinate corresponding to the first image coordinate point and the first three-dimensional coordinate point; calculate a third projection coordinate obtained by projecting the first three-dimensional coordinate point on the standardized sphere based on the initial value of the pose of the two-dimensional camera and the first three-dimensional coordinate point; calculate a fourth projection coordinate obtained by projecting the second three-dimensional coordinate point on the standardized sphere based on the initial value of the pose of the two-dimensional camera and the second three-dimensional coordinate point.

[0186] The projection module 202 is specifically configured to calculate a product of the initial value of the pose of the two-dimensional camera and a coordinate vector of the first three-dimensional coordinate point, to obtain a target coordinate point corresponding to the first three-dimensional coordinate point in the camera coordinate system of the two-dimensional camera; take a norm of a coordinate vector of the target coordinate point, calculate a ratio between the coordinate vector of the target coordinate point and the norm, to obtain the third projection coordinate of the first three-dimensional coordinate point on the standardized sphere.

[0187] The pose calculation module 203 is specifically configured to calculate a first total projection error corresponding to the two-dimensional camera and the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate; optimize the first total projection error to obtain a value of a target pose of the two-dimensional camera when the first total projection error takes a minimum value; calculate the pose of the depth camera based on the preset conversion matrix and the value of the target pose of the two-dimensional camera.

[0188] The pose calculation module 203 is configured to calculate a difference between the first projection coordinate corresponding to the first image coordinate point and the third projection coordinate of the first three-dimensional coordinate point corresponding to the first image coordinate point, to obtain a projection error corresponding to the first three-dimensional coordinate point; calculate a difference between the second projection coordinate corresponding to the second image coordinate point and the fourth projection coordinate of the second three-dimensional coordinate point corresponding to the second image coordinate point, to obtain a projection error corresponding to the second three-dimensional coordinate point; calculate a first total projection error corresponding to the two-dimensional camera and the depth camera based on the projection error corresponding to the first three-dimensional coordinate point and the projection error corresponding to the second three-dimensional coordinate point.

[0189] The pose calculation module 203 is specifically configured to calculate an initial value of a pose of the two-dimensional camera based on the first projection coordinates corresponding to the first image coordinate point and the first three-dimensional coordinate point; calculate a first Jacobian matrix of a projection error of the first three-dimensional coordinate point with respect to the initial value of the pose of the two-dimensional camera based on the initial value of the pose of the two-dimensional camera, the first three-dimensional coordinate point, and the projection error corresponding to the first three-dimensional coordinate point; calculate a first Jacobian matrix of a projection error of the second three-dimensional coordinate point with respect to the initial value of the pose of the two-dimensional camera based on the initial value of the pose of the two-dimensional camera, the second three-dimensional coordinate point, and the projection error corresponding to the second three-dimensional coordinate point; and optimize the first total projection error by using a preset optimization algorithm based on the initial value of the pose of the two-dimensional camera, the first total projection error, the first Jacobian matrix corresponding to the first three-dimensional coordinate point, and the first Jacobian matrix corresponding to the second three-dimensional coordinate point.

[0190] In some embodiments of the present application, during the movement of the two-dimensional camera and the depth camera, the two-dimensional camera captures a plurality of two-dimensional images of the real scene at a plurality of time points, and the depth camera captures a plurality of depth images of the real scene at a plurality of time points; based on the plurality of two-dimensional images and the plurality of depth images, the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinates, the second projection coordinates, the third projection coordinates, and the fourth projection coordinates corresponding to each time point in the plurality of time points are obtained.

[0191] The pose calculation module 203 is specifically configured to calculate a second total projection error corresponding to the two-dimensional camera and the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinates, the second projection coordinates, the third projection coordinates, and the fourth projection coordinates corresponding to each time point; optimize the second total projection error to obtain a target pose value of the two-dimensional camera corresponding to each time point when the second total projection error takes a minimum value; and calculate the pose of the depth camera at each time point based on a preset conversion matrix and the target pose value of the two-dimensional camera corresponding to each time point.

[0192] The pose calculation module 203 is specifically configured to calculate a first Jacobian matrix of the projection error of the first three-dimensional coordinate point at the first time to the initial value of the pose of the two-dimensional camera at the first time based on the initial value of the pose of the two-dimensional camera at the first time, the first three-dimensional coordinate point captured by the two-dimensional camera at the first time, and the projection error corresponding to the first three-dimensional coordinate point at the first time; the first time is any time in the plurality of times; calculate a first Jacobian matrix of the projection error of the second three-dimensional coordinate point at the first time to the initial value of the pose of the two-dimensional camera at the first time based on the initial value of the pose of the two-dimensional camera at the first time, the second three-dimensional coordinate point captured by the depth camera at the first time, and the projection error corresponding to the second three-dimensional coordinate point at the first time; and optimize the second total projection error by a preset optimization algorithm based on the initial value of the pose of the two-dimensional camera at each time, the second total projection error, the first Jacobian matrix corresponding to the first three-dimensional coordinate point at each time, and the first Jacobian matrix corresponding to the second three-dimensional coordinate point at each time.

[0193] The pose calculation module 203 is specifically configured to calculate a first Jacobian matrix of the projection error of the first three-dimensional coordinate point at the first time to the initial value of the pose of the two-dimensional camera at the first time based on the initial value of the pose of the two-dimensional camera at the first time, the first three-dimensional coordinate point captured by the two-dimensional camera at the first time, and the projection error corresponding to the first three-dimensional coordinate point at the first time; the first time is any time in the plurality of times; calculate a first Jacobian matrix of the projection error of the second three-dimensional coordinate point at the first time to the initial value of the pose of the two-dimensional camera at the first time based on the initial value of the pose of the two-dimensional camera at the first time, the second three-dimensional coordinate point captured by the depth camera at the first time, and the projection error corresponding to the second three-dimensional coordinate point at the first time; and optimize the second total projection error by a preset optimization algorithm based on the initial value of the pose of the two-dimensional camera at each time, the second total projection error, the first Jacobian matrix corresponding to the first three-dimensional coordinate point at each time, and the first Jacobian matrix corresponding to the second three-dimensional coordinate point at each time; and update the coordinates of the corresponding three-dimensional coordinate points on the map constructed during the movement of the two-dimensional camera and the depth camera based on the coordinate values of the first three-dimensional coordinate point and the second three-dimensional coordinate point at each time when the second total projection error takes the minimum value.

[0194] The camera pose calculation device based on multi-type image processing provided by the embodiments of the present application and the camera pose calculation method based on multi-type image processing provided by the above embodiments have the same beneficial effects as the method they adopt, run or implement.

[0195] The embodiments of the present application also provide a camera pose calculation system, which is shown in Figure 7 The system includes a visible light camera 1, a depth camera 2, and a processor 3. The processor 3 is in communication connection with the visible light camera 1 and the depth camera 2 respectively.

[0196] The visible light camera 1 is configured to capture a visible light image of a real scene.

[0197] a depth camera 2 configured to capture a depth image of the real scene;

[0198] a processor 3 configured to obtain a first image coordinate point corresponding to a first three-dimensional coordinate point in the real scene from a visible light image captured by the visible light camera 1, obtain a second image coordinate point corresponding to a second three-dimensional coordinate point in the real scene from a depth image captured by the depth camera 2, project the first image coordinate point and the second image coordinate point onto a normalized sphere to obtain a first projection coordinate corresponding to the first image coordinate point and a second projection coordinate corresponding to the second image coordinate point, respectively, project the first three-dimensional coordinate point and the second three-dimensional coordinate point onto the normalized sphere to obtain a third projection coordinate corresponding to the first three-dimensional coordinate point and a fourth projection coordinate corresponding to the second three-dimensional coordinate point, respectively, and calculate a pose of the visible light camera 1 and a pose of the depth camera 2 based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate, and the fourth projection coordinate, wherein a sphere center of the normalized sphere is an optical center of the visible light camera 1.

[0199] The specific operation details of the visible light camera 1, the depth camera 2, and the processor 3 in the system can refer to the operation details of the visible light camera 1, the depth camera 2, and the processor 3 in the camera pose calculation method based on multi-type image processing provided in the above embodiments, which will not be described here.

[0200] The camera pose calculation system provided in the embodiments of the present application and the camera pose calculation method based on multi-type image processing provided in the above embodiments have the same beneficial effects as the method adopted, run, or implemented.

[0201] The embodiments of the present application also provide an electronic device corresponding to the camera pose calculation method based on multi-type image processing provided in the above embodiments. Please refer to Figure 8 , which shows a schematic diagram of an electronic device provided in some embodiments of the present application. As Figure 8 shown, the electronic device 30 can include a processor 300, a memory 301, a bus 302, and a communication interface 303, wherein the processor 300, the communication interface 303, and the memory 301 are connected through the bus 302; the memory 301 stores a computer program executable on the processor 300, and the processor 300 executes the computer program to perform the camera pose calculation method based on multi-type image processing provided in any of the preceding embodiments of the present application.

[0202] The memory 301 can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port 303 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0203] The bus 302 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs, and the processor 300 executes the programs after receiving execution instructions. The camera pose calculation method based on multi-class image processing disclosed in any of the embodiments of the present application can be applied to the processor 300 or implemented by the processor 300.

[0204] The processor 300 can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 300 or the instructions in the form of software. The processor 300 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-program gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 301, and the processor 300 reads the information in the memory 301 and combines the hardware to complete the steps of the above method.

[0205] The electronic device provided by the embodiments of the present application and the camera pose calculation method based on multi-class image processing provided by the embodiments of the present application have the same beneficial effects as the method they adopt, run or implement.

[0206] The embodiment of the present application further provides a computer readable storage medium corresponding to the camera pose calculation method based on multi-class image processing provided in the foregoing embodiment, and a computer program (i.e., a program product) is stored on the computer readable storage medium, and the computer program is executed by a processor to perform the camera pose calculation method based on multi-class image processing provided in any of the foregoing embodiments.

[0207] It should be noted that examples of the computer readable storage medium can further include, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, or other optical, magnetic storage medium, which will not be listed one by one here.

[0208] The embodiment of the present application further provides a computer program product corresponding to the camera pose calculation method based on multi-class image processing provided in the foregoing embodiment, and the computer program product comprises a computer program executed by a processor to implement the camera pose calculation method based on multi-class image processing provided in the foregoing embodiments.

[0209] The computer readable storage medium and the computer program product provided in the foregoing embodiments of the present application are based on the same inventive concept as the camera pose calculation method based on multi-class image processing provided in the embodiments of the present application, and have the same beneficial effects as the method adopted, executed or implemented by the application program stored therein.

[0210] It should be noted that:

[0211] The algorithms and displays provided herein are not inherently related to any particular computer, virtual apparatus, or other device. Various general purpose devices can be used with these teachings based on the description herein. The construction required to construct such devices as set forth above is within the skill of one in the relevant art. In addition, the present application is not restricted to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the application as described herein, and any references below to specific languages are provided for disclosure of enablement only.

[0212] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known methods, structures and techniques are not described in detail in order not to obscure the understanding of the specification.

[0213] Similarly, it is to be understood that the embodiments of the present application can be alternately or additionally employed in combination with each other, where appropriate, even though not expressly stated in the above description. Although the present application has been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations to the embodiments and modifications of the present application, such as: to the embodiments of the present application, to the methods of the present application, and to the devices of the present application. Accordingly, many modifications can be made by one of ordinary skill in the art without departing from the scope of the present application as disclosed herein. For example, although processes are described with regard to particular operational flows, various other processes can be implemented as steps within a larger process that includes the described steps, or as at least partially parallel processes that are performed concurrently with the described processes. Accordingly, other implementations are within the scope of the following claims.

[0214] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be divided into more sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and any method or of the device disclosed in the specification in combination with all the features disclosed in the specification (including the accompanying claims, abstract and drawings) can be taken, except that at least some of such features and / or processes or units are mutually exclusive. Unless explicitly stated, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features that serve the same, equivalent or similar purpose.

[0215] Further, those skilled in the art will appreciate that a combination of features of different embodiments means within the scope of the present application and forms different embodiments, although some embodiments described herein include certain features rather than others included in other embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0216] The various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. As will be appreciated by one skilled in the art, a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some or all of the components of the virtual machine creation apparatus according to embodiments of the present application. The present application can also be implemented as a program of instructions for performing part or all of the methods described herein, e.g., a computer program and a computer program product. Such program of the present application can be stored on a computer readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier medium, or in any other form.

[0217] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unitary claim, several devices, apparatuses or means can be listed, comprising means for performing a certain function. The use of the term'means' in a claim is intended to cover one or more elements of a device, apparatus or means for performing the function in question. None of the elements recited in the claims are intended to be impliedly disclosed away. The use of the words 'first','second' and 'third' and the like in the description does not imply any order.

[0218] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this, and any changes or replacements within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A camera pose calculation method based on multi-class image processing, characterized in that, The method comprises the following steps: obtaining a first image coordinate point corresponding to a first three-dimensional coordinate point in a real scene from a two-dimensional image captured by a two-dimensional camera; obtaining a second image coordinate point corresponding to a second three-dimensional coordinate point in the real scene from a depth image captured by a depth camera; projecting the first image coordinate point and the second image coordinate point onto a standardized spherical surface to obtain a first projection coordinate corresponding to the first image coordinate point and a second projection coordinate corresponding to the second image coordinate point, respectively; the center of the standardized spherical surface is the optical center of the two-dimensional camera; projecting the first three-dimensional coordinate point and the second three-dimensional coordinate point onto the standardized spherical surface to obtain a third projection coordinate corresponding to the first three-dimensional coordinate point and a fourth projection coordinate corresponding to the second three-dimensional coordinate point, respectively; calculating the pose of the two-dimensional camera and the pose of the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate; wherein the calculation of the pose of the two-dimensional camera and the pose of the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate comprises: calculating a first total projection error corresponding to the two-dimensional camera and the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate; optimizing the first total projection error to obtain a target pose value of the two-dimensional camera when the first total projection error takes a minimum value; calculating the pose of the depth camera based on a preset conversion matrix and the target pose value; the preset conversion matrix is a conversion matrix between the camera coordinate system of the two-dimensional camera and the camera coordinate system of the depth camera.

2. The method of claim 1, wherein, projecting the second image coordinate point onto the standardized spherical surface to obtain the second projection coordinate corresponding to the second image coordinate point, comprising: calculating a plane projection coordinate of a pixel coordinate on a standardized plane based on the camera internal parameter of the depth camera and the pixel coordinate and the depth included in the second image coordinate point; the standardized plane is a tangent plane of the standardized spherical surface; calculating a pixel projection coordinate of the pixel coordinate projected onto the standardized spherical surface and a projection depth of the depth projected onto the standardized spherical surface based on the depth, the plane projection coordinate and a preset conversion matrix; the preset conversion matrix is a conversion matrix between the camera coordinate system of the two-dimensional camera and the camera coordinate system of the depth camera; determining the pixel projection coordinate and the projection depth as the second projection coordinate corresponding to the second image coordinate point.

3. The method of claim 1, wherein, the projecting the first three-dimensional coordinate point and the second three-dimensional coordinate point onto the standardized spherical surface to obtain the third projection coordinate corresponding to the first three-dimensional coordinate point and the fourth projection coordinate corresponding to the second three-dimensional coordinate point, comprising: calculating, based on the first projection coordinate corresponding to the first image coordinate point and the first three-dimensional coordinate point, an initial value of a pose of the two-dimensional camera; calculating, based on the initial value of the pose and the first three-dimensional coordinate point, a third projection coordinate obtained by projecting the first three-dimensional coordinate point onto the standardized spherical surface; calculating, based on the initial value of the pose and the second three-dimensional coordinate point, a fourth projection coordinate obtained by projecting the second three-dimensional coordinate point onto the standardized spherical surface.

4. The method of claim 3, wherein, The method further includes the following steps. calculating a product of the initial value of the pose and a coordinate vector of the first three-dimensional coordinate point to obtain a target coordinate point corresponding to the first three-dimensional coordinate point in a camera coordinate system of the two-dimensional camera; calculating a ratio between a coordinate vector of the target coordinate point and a norm of the coordinate vector of the target coordinate point to obtain the third projection coordinate of the first three-dimensional coordinate point on the standardized spherical surface.

5. The method of claim 1, wherein, The method further includes the following steps. calculating a difference between the first projection coordinate corresponding to the first image coordinate point and the third projection coordinate of the first three-dimensional coordinate point corresponding to the first image coordinate point to obtain a projection error corresponding to the first three-dimensional coordinate point; calculating a difference between the second projection coordinate corresponding to the second image coordinate point and the fourth projection coordinate of the second three-dimensional coordinate point corresponding to the second image coordinate point to obtain a projection error corresponding to the second three-dimensional coordinate point; calculating the first total projection error corresponding to the two-dimensional camera and the depth camera based on the projection error corresponding to the first three-dimensional coordinate point and the projection error corresponding to the second three-dimensional coordinate point.

6. The method of claim 5, wherein, The method further includes the following steps. calculating, based on the first projection coordinate corresponding to the first image coordinate point and the first three-dimensional coordinate point, an initial value of a pose of the two-dimensional camera; calculating, based on the initial value of the pose, the first three-dimensional coordinate point, and the projection error corresponding to the first three-dimensional coordinate point, a first Jacobian matrix of the projection error of the first three-dimensional coordinate point with respect to the initial value of the pose; calculating, based on the initial value of the pose, the second three-dimensional coordinate point, and the projection error corresponding to the second three-dimensional coordinate point, a first Jacobian matrix of the projection error of the second three-dimensional coordinate point with respect to the initial value of the pose; optimizing the first total projection error by a preset optimization algorithm based on the initial value of the pose, the first total projection error, the first Jacobian matrix corresponding to the first three-dimensional coordinate point, and the first Jacobian matrix corresponding to the second three-dimensional coordinate point.

7. The method of claim 1, wherein, During the movement of the two-dimensional camera and the depth camera, the two-dimensional camera captures a plurality of two-dimensional images of a real scene at a plurality of time points, and the depth camera captures a plurality of depth images of the real scene at the plurality of time points; based on the plurality of two-dimensional images and the plurality of depth images, the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate corresponding to each of the plurality of time points are obtained; based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate, the pose of the two-dimensional camera and the pose of the depth camera are calculated, comprising: based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate and the fourth projection coordinate corresponding to each time point, the second total projection error corresponding to the two-dimensional camera and the depth camera is calculated; the second total projection error is optimized to obtain the target pose value of the two-dimensional camera corresponding to each time point when the second total projection error takes the minimum value; based on the preset conversion matrix and the target pose value of the two-dimensional camera corresponding to each time point, the pose of the depth camera at each time point is calculated; the preset conversion matrix is the conversion matrix between the camera coordinate system of the two-dimensional camera and the camera coordinate system of the depth camera.

8. The method of claim 7, wherein, the optimization of the second total projection error comprises: based on the initial value of the pose of the two-dimensional camera at the first time, the first three-dimensional coordinate point captured by the two-dimensional camera at the first time and the projection error corresponding to the first three-dimensional coordinate point at the first time, the first Jacobian matrix of the projection error of the first three-dimensional coordinate point at the first time to the initial value of the pose of the two-dimensional camera at the first time is calculated; the first time is any time in the plurality of time points; based on the initial value of the pose of the two-dimensional camera at the first time, the second three-dimensional coordinate point captured by the depth camera at the first time and the projection error corresponding to the second three-dimensional coordinate point at the first time, the first Jacobian matrix of the projection error of the second three-dimensional coordinate point at the first time to the initial value of the pose of the two-dimensional camera at the first time is calculated; based on the initial value of the pose of the two-dimensional camera at each time, the second total projection error, the first Jacobian matrix corresponding to the first three-dimensional coordinate point at each time and the first Jacobian matrix corresponding to the second three-dimensional coordinate point at each time, the second total projection error is optimized by a preset optimization algorithm.

9. The method of claim 8, wherein, the method further comprises: based on the initial value of the pose of the two-dimensional camera at the first time, the first three-dimensional coordinate point captured by the two-dimensional camera at the first time and the projection error corresponding to the first three-dimensional coordinate point at the first time, the second Jacobian matrix of the projection error of the first three-dimensional coordinate point at the first time to the first three-dimensional coordinate point at the first time is calculated; The first Jacobian matrix of the projection error of the second three-dimensional coordinate point at the first moment based on the initial value of the pose of the two-dimensional camera at the first moment, the second three-dimensional coordinate point captured by the depth camera at the first moment, and the projection error corresponding to the second three-dimensional coordinate point at the first moment; The second total projection error is optimized by a preset optimization algorithm based on the initial value of the pose of the two-dimensional camera at each moment, the second total projection error, and the first Jacobian matrix and the second Jacobian matrix at each moment; The coordinates of the corresponding three-dimensional coordinate points on the map constructed during the movement of the two-dimensional camera and the depth camera are updated based on the coordinate values of the first three-dimensional coordinate point and the second three-dimensional coordinate point at each moment when the second total projection error takes the minimum value.

10. A pose computation system of a video camera, characterized by, It comprises: a visible light camera configured to capture a visible light image of a real scene; a depth camera configured to capture a depth image of the real scene; a processor configured to obtain a first image coordinate point corresponding to a first three-dimensional coordinate point in the real scene from the visible light image captured by the visible light camera; obtain a second image coordinate point corresponding to a second three-dimensional coordinate point in the real scene from the depth image captured by the depth camera; project the first image coordinate point and the second image coordinate point onto a standardized sphere to obtain a first projection coordinate corresponding to the first image coordinate point and a second projection coordinate corresponding to the second image coordinate point, respectively; project the first three-dimensional coordinate point and the second three-dimensional coordinate point onto the standardized sphere to obtain a third projection coordinate corresponding to the first three-dimensional coordinate point and a fourth projection coordinate corresponding to the second three-dimensional coordinate point, respectively; and calculate the pose of the visible light camera and the pose of the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate, and the fourth projection coordinate; wherein the center of the standardized sphere is the optical center of the visible light camera; wherein the calculation of the pose of the visible light camera and the pose of the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate, and the fourth projection coordinate comprises: calculating a first total projection error corresponding to the visible light camera and the depth camera based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the first projection coordinate, the second projection coordinate, the third projection coordinate, and the fourth projection coordinate; optimizing the first total projection error to obtain a target pose value of the visible light camera when the first total projection error takes the minimum value; and calculating the pose of the depth camera based on a preset conversion matrix and the target pose value; the preset conversion matrix is a conversion matrix between the camera coordinate system of the visible light camera and the camera coordinate system of the depth camera.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-9.

12. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by a processor to implement the method of any one of claims 1-9.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method of any one of claims 1-9.

Images