Position calculation method and device, electronic device, and readable storage medium

Abstract

The present application relates to a posture calculation method and device, an electronic device, and a computer-readable storage medium. If a depth image of the current environment is first collected within a preset initialization sliding window, the posture of the electronic device when collecting the depth image is determined as the initial posture; wherein the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window. According to the initial posture, the target RGB image, the depth image, and the next frame RGB image of the target RGB image, the posture of the electronic device when collecting the next frame RGB image is determined. Therefore, by using the posture calculation method in the present application, it is not necessary to collect a full RGB image within the preset initialization sliding window before the electronic device can output the posture when collecting the first frame RGB image outside the preset initialization sliding window. The posture can be output in real time within the preset initialization sliding window, which reduces the waiting time for real-time output of the posture.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a posture calculation method and device, an electronic device, and a readable storage medium. Background Art

[0002] The position and posture (abbreviated as posture) of electronic devices in unknown environments is one of the key technologies in industries such as augmented reality, virtual reality, mobile robots, and unmanned driving. With the rapid development of these industries, the accuracy of electronic devices in locating objects in the surrounding environment has also been increasingly required. When using the VIO (Visual-Inertial Odometry) system to locate objects in the surrounding environment, one of the core steps is to determine the posture of the electronic device when taking each frame of the image.

[0003] Traditional methods, when determining the posture of an electronic device when taking each frame of an image, need to collect a certain number of RGB images or depth images of the surrounding environment based on the camera on the electronic device, so that it can be initialized based on the certain number of RGB images or depth images, and then output the posture of the electronic device when taking each frame of an image in real time. Obviously, it takes a long time to collect a certain number of RGB images and / or depth images, which leads to a long waiting time for real-time output of the posture. Summary of the invention

[0004] The embodiments of the present application provide a posture calculation method and device, an electronic device, and a readable storage medium, which can reduce the waiting time for real-time output of posture.

[0005] A posture calculation method, the method comprising:

[0006] If a depth image of the current environment is collected for the first time within a preset initialization sliding window, the posture of the electronic device when collecting the depth image is determined as the initial posture; wherein the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window;

[0007] The posture of the electronic device when collecting the next frame of RGB image is determined according to the initial posture, the target RGB image, the depth image and the next frame of RGB image of the target RGB image.

[0008] A posture calculation device, the device comprising:

[0009] An initial posture determination module is used to determine the posture of the electronic device when collecting the depth image as the initial posture if a depth image of the current environment is collected for the first time within a preset initialization sliding window; wherein the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window;

[0010] The posture determination module is used to determine the posture of the electronic device when collecting the next frame of RGB image according to the initial posture, the target RGB image, the depth image and the next frame of RGB image of the target RGB image.

[0011] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the posture calculation method as described above.

[0012] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the posture calculation method as described above.

[0013] The posture calculation method and device, electronic device, and computer-readable storage medium, if the depth image of the current environment is first collected within the preset initialization sliding window, the posture of the electronic device when collecting the depth image is determined as the initial posture; wherein the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window. According to the initial posture, the target RGB image, the depth image, and the next frame RGB image of the target RGB image, the posture of the electronic device when collecting the next frame RGB image is determined. Therefore, by adopting the posture calculation method in the present application, it is not necessary to collect a full RGB image within the preset initialization sliding window before the electronic device can output the posture when collecting the first frame RGB image outside the preset initialization sliding window. Instead, when the depth image of the current environment is first collected within the preset initialization sliding window, and the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window, the posture of the electronic device when collecting the depth image can be determined as the initial posture, the target RGB image, the depth image, and the next frame RGB image of the target RGB image, and the posture of the electronic device when collecting the next frame RGB image is determined. Obviously, the posture can be output in real time within the preset initialization sliding window, which reduces the waiting time for real-time output of the posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A diagram showing an application environment of a posture calculation method in an embodiment;

[0016] Figure 2 is a flow chart of a method for calculating a posture in one embodiment;

[0017] Figure 3 is a flow chart of a method for calculating a posture in another embodiment;

[0018] Figure 4 A schematic diagram of constructing a target three-dimensional map of the current environment if the target RGB image is not the first frame image in a preset initialization sliding window in an embodiment;

[0019] Figure 5 A flowchart of a method for calculating the first position transformation matrix between a next frame RGB image and a target RGB image using a preset perspective projection PnP algorithm in one embodiment;

[0020] Figure 6 A schematic diagram of calculating a posture transformation matrix using a preset perspective projection PnP algorithm in one embodiment;

[0021] Figure 7 for Figure 5 A flowchart of a method for calculating a first translation transformation matrix between a next frame RGB image and a target RGB image;

[0022] Figure 8 It is a flowchart of a method for constructing a target three-dimensional map of a current environment according to a first pose transformation matrix and a depth image in one embodiment;

[0023] Fig. 9 for Figure 3 A flowchart of a method for calculating the position and posture of an electronic device when collecting an RGB image after collecting a next frame of RGB image within a preset initialization sliding window according to a target three-dimensional map;

[0024] Fig.10 A schematic diagram of calculating the pose and three-dimensional map when calculating an RGB image after acquiring the next RGB image frame in one embodiment;

[0025] Fig.11 A flowchart of a method for calculating a posture in another embodiment;

[0026] Fig.12 is a schematic diagram of a posture calculation method in yet another embodiment;

[0027] Fig.13 It is a flowchart of a method for calculating a posture in a specific embodiment;

[0028] Fig.14 is a structural block diagram of a posture calculation device in one embodiment;

[0029] Fig.15 is a structural block diagram of a posture calculation device in one embodiment;

[0030] Fig.16 is a structural block diagram of a posture calculation device in another embodiment;

[0031] Fig.17 FIG. 1 is a schematic diagram of the internal structure of an electronic device in an embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] The position and posture (abbreviated as posture) of electronic devices in unknown environments is one of the key technologies in industries such as augmented reality, virtual reality, mobile robots, and unmanned driving. With the rapid development of these industries, the accuracy of electronic devices in locating objects in the surrounding environment has also been increasingly required. When using the VIO (Visual-Inertial Odometry) system to locate objects in the surrounding environment, one of the core steps is to determine the posture of the electronic device when taking each frame of the image.

[0034] One of the traditional methods uses the VINS (visual-inertial system)-MONO algorithm to determine the pose of the electronic device when taking each frame of image. The specific steps of the VINS-MONO algorithm include: assuming that there are 10 frames of images in the preset initialization sliding window. Of course, the size of the preset initialization sliding window is not specifically limited in this application. Then, the first step is to collect RGB images through the camera on the electronic device. After 10 frames of RGB images are accumulated in the preset initialization sliding window, two frames of images (such as L frames and R frames) with parallax conditions that meet the conditions are selected from the 10 frames of RGB images, and then the epipolar geometry constraints are used to calculate the pose between the two frames. The second step is to use the triangulation method to restore the map points that are co-visible between the two frames using the pose. The third step is to project these map points onto any frame of the above 10 frames of RGB images except the L frame and the R frame, and calculate the pose of any frame by minimizing the reprojection error. In the fourth step, the triangulation method is used between any frame and L frame and R frame to restore the map points that are co-visible between any frame and L frame and R frame. By repeating the third and fourth steps, the poses of the 10 frames of RGB images in the preset initialization sliding window and the map points corresponding to these 10 frames of RGB images can be calculated.

[0035] There is another traditional method, which uses the VINS-RGBD algorithm to determine the pose of the electronic device when taking each frame of the image. The specific steps of the VINS-RGBD algorithm include: Assuming that there are 10 frames of images in the preset initialization sliding window, then, in the first step, the RGB image and the depth image are collected simultaneously by the camera on the electronic device, and it is necessary to ensure that each frame of the RGB image has a corresponding depth image. The correspondence here means that the RGB image is aligned with the depth image in time and space. In the second step, after accumulating 10 frames of RGB images and corresponding depth images in the preset initialization sliding window, two frames of images (such as L frame and R frame) whose parallax meets the conditions are selected from these 10 frames of RGB images. The pose of the L frame is calculated simultaneously using the traditional PnP algorithm, and then combined with the depth image corresponding to the L frame, the reprojection error is used to meet the preset threshold to select valid map points, and then the map points are restored. In the third step, these map points are projected onto any frame of the other frames in the above 10 frames of RGB images except the L frame and the R frame, and the pose of any frame is calculated separately by minimizing the reprojection error. The fourth step is to use the depth image of any frame to restore the map points that can be co-visible with the L frame and whose reprojection error meets the threshold requirement. Repeat the third and fourth steps to calculate the poses of the 10 RGB images in the preset initialization sliding window and the map points corresponding to these 10 RGB images.

[0036] The above two traditional methods, when outputting the posture in real time, both need to collect a certain number of RGB images or depth images of the surrounding environment based on the camera on the electronic device, so that the posture of the electronic device when taking each frame of image can be output in real time based on the certain number of RGB images or depth images. Obviously, it takes a long time to collect a certain number of RGB images and / or depth images, which leads to a long waiting time for real-time output of posture.

[0037] Therefore, in order to solve the problem of long waiting time for real-time output of posture, a posture calculation method is proposed in an embodiment of the present application. It is not necessary to collect a full RGB image within the preset initialization sliding window before the electronic device can output the posture when collecting the first frame of RGB image outside the preset initialization sliding window. Instead, when the depth image of the current environment is first collected within the preset initialization sliding window, and the target RGB image corresponding to the depth image is not the last frame of image within the preset initialization sliding window, the posture of the electronic device when collecting the depth image can be determined as the initial posture, target RGB image, depth image and the next frame of RGB image of the target RGB image, and the posture of the electronic device when collecting the next frame of RGB image is determined. Obviously, the posture can be output in real time within the preset initialization sliding window, which reduces the waiting time for real-time output of the posture.

[0038] Figure 1 FIG. 1 is an application scenario diagram of a posture calculation method in an embodiment. Figure 1 As shown, the application environment includes an electronic device 120, and the electronic device 120 includes a first camera and a second camera. Among them, the first camera is an RGB camera, and the second camera is a camera for collecting depth images, for example, a TOF (Time-offlight, time of flight) camera or a structured light camera, etc., which is not limited in this application. The electronic device 120 collects the RGB image and the depth image of the current environment through the first camera and the second camera respectively. If the electronic device 120 collects the depth image of the current environment for the first time within the preset initialization sliding window, the posture of the electronic device when collecting the depth image is determined as the initial posture; wherein, the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window. According to the initial posture, the target RGB image, the depth image and the next frame RGB image of the target RGB image, the posture of the electronic device when collecting the next frame RGB image is determined. The electronic device 120 can be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a wearable device (smart bracelet, smart watch, smart glasses, smart gloves, smart socks, smart belt, etc.), a VR (virtual reality) device, a smart home, a driverless car, etc.

[0039] Figure 2 FIG. 1 is a flowchart of a posture calculation method in an embodiment. The posture calculation method in this embodiment is executed on Figure 1 The electronic device 120 in the embodiment is described as an example, and the electronic device 120 includes a first camera and a second camera. The first camera is an RGB camera, and the second camera is a camera for collecting depth images. The posture calculation method includes steps 220 to 240, wherein:

[0040] Step 220, if the depth image of the current environment is collected for the first time within the preset initialization sliding window, the posture of the electronic device when collecting the depth image is determined as the initial posture; wherein the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window.

[0041] In the traditional method, it is necessary to collect a certain number of image frames before the position and posture of the electronic device when taking each frame of image can be output in real time. These certain number of image frames fill up the preset initialization sliding window (hereinafter referred to as sliding window), that is, the size of the preset initialization sliding window is equal to the duration of collecting a certain number of image frames. For example, if the preset initialization sliding window includes 10 frames of RGB images, then after collecting 10 frames of RGB images, these 10 frames of RGB images fill up the preset initialization sliding window, and the size of the preset initialization sliding window is equal to the duration of collecting 10 frames of RGB images.

[0042] In an embodiment of the present application, it is not necessary to collect a full RGB image within the preset initialization sliding window before the electronic device can output the posture when collecting the first frame of RGB image outside the preset initialization sliding window. Instead, the RGB image is collected by the first camera in the electronic device, and the depth image is collected by the second camera at the same time. If the depth image of the current environment is collected for the first time within the preset initialization sliding window, and the target RGB image corresponding to the depth image is not the last frame image in the preset initialization sliding window, the posture of the electronic device when collecting the depth image is determined as the initial posture, and the image coordinate system of the target RGB image is used as the world coordinate system, thereby realizing the visual initialization of the electronic device. And there is no need to filter out two frames of images with parallax that meet the conditions from these 10 frames of RGB images, so it has wider adaptability.

[0043] First, since the acquisition frequency of the depth image acquired by the second camera may be lower than the acquisition frequency of the RGB image acquired by the first camera, it will result in the failure to acquire the depth image corresponding to each frame of the RGB image, that is, part of the depth image will be missing relative to the RGB image. Second, in electronic devices, due to abnormalities in the second camera and related hardware and software, part of the depth image will be missing relative to the RGB image or the alignment of the acquired RGB image and the depth image will be abnormal. All of the above will result in the failure to acquire the corresponding depth image for each frame of the RGB image.

[0044] Step 240, determining the posture of the electronic device when capturing the next frame of RGB image based on the initial posture, the target RGB image, the depth image and the next frame of RGB image of the target RGB image.

[0045] The posture of the electronic device when collecting the depth image is determined as the initial posture, and the image coordinate system of the target RGB image is used as the world coordinate system. After the visual initialization of the electronic device is achieved, the posture when collecting the next frame of RGB image can be output in real time.

[0046] Specifically, after determining the initial posture and world coordinate system, the posture of the electronic device when collecting the next frame of RGB image is determined according to the target RGB image, the depth image and the next frame of RGB image of the target RGB image. First, the perspective projection PnP algorithm can be used, based on the 2D-2D feature point pairs that match each other between the target RGB image and the next frame of RGB image of the target RGB image; secondly, according to these 2D-2D feature point pairs combined with the 3D points on the depth image, the matching 3D-2D feature point pairs are obtained; again, based on the 3D-2D feature point pairs, the posture transformation matrix between the next frame of RGB image of the target RGB image and the target RGB image is calculated. Finally, based on the posture transformation matrix and the initial posture, the posture of the next frame of RGB image can be generated. Among them, posture refers to position and attitude, and posture is a six-dimensional matrix containing three positions (X, Y, Z) and three attitude angles (heading, pitch, roll).

[0047] The perspective projection PnP algorithm here can be a traditional perspective projection PnP algorithm, that is, based on the 3D-2D feature point pairs, the rotation transformation matrix and the translation transformation matrix between the two frames are simultaneously calculated, and the rotation transformation matrix and the translation transformation matrix constitute the posture transformation matrix. Of course, the perspective projection PnP algorithm here can be a new perspective projection PnP algorithm, that is, based on the 3D-2D feature point pairs, the rotation transformation matrix and the translation transformation matrix between the two frames are calculated step by step.

[0048] In an embodiment of the present application, it is not necessary to collect a full RGB image within the preset initialization sliding window before the electronic device can output the posture when collecting the first frame of RGB image outside the preset initialization sliding window. Instead, when the depth image of the current environment is first collected within the preset initialization sliding window, and the target RGB image corresponding to the depth image is not the last frame of image within the preset initialization sliding window, the posture of the electronic device when collecting the depth image can be determined as the initial posture, target RGB image, depth image and the next frame of RGB image of the target RGB image, and the posture of the electronic device when collecting the next frame of RGB image is determined. The time for the first output of the posture is greatly advanced from outside the preset initialization sliding window to inside the sliding window and to the next frame when the depth image is first received. Thus, the posture can be output in real time within the preset initialization sliding window, reducing the waiting time for real-time output of the posture.

[0049] At the same time, it is not necessary to collect a depth image corresponding to each frame of RGB image before the pose can be output in real time. Instead, a frequency lower than the frequency of collecting RGB images can be used to collect depth maps, or a variable frequency can be used to collect depth maps, and as long as the depth map is collected within the sliding window, initialization can be started, and the pose can be output in real time after initialization. Since the depth map can be collected at a lower frequency or a variable frequency, the pose can be initialized and output in real time based on the low-frequency depth map, thereby avoiding the collection of a large amount of data and the processing of a large amount of data, thereby reducing the power consumption of electronic devices.

[0050] Continuing from the previous embodiment, after calculating the posture of the electronic device when collecting the next frame of RGB image, the posture of the electronic device when collecting other RGB images located after the next frame of RGB image within the preset initialization sliding window is calculated in sequence. Then, if Figure 3 As shown, a posture calculation method is provided, which also includes:

[0051] Step 260, constructing a target three-dimensional map of the current environment according to the posture and depth image when the electronic device collects the next frame of RGB image.

[0052] In step 240, the position and posture of the electronic device when capturing the next frame of RGB image is calculated. At this time, the target three-dimensional map of the current environment can be constructed based on the depth image of the target RGB image and the position and posture of the next frame of RGB image. There are two cases when constructing the target three-dimensional map of the current environment.

[0053] One of the situations is that, assuming that the target RGB image is the first frame image in the preset initialization sliding window, that is, for the first frame RGB image in the preset initialization sliding window, the corresponding depth image is collected. Then, because there is no RGB image before the target RGB image, when constructing the target three-dimensional map of the current environment, it is only necessary to calculate the first position transformation matrix between the next frame RGB image and the target RGB image. According to the first position transformation matrix and the depth image, the target three-dimensional map of the current environment can be constructed. Here, if the first position transformation matrix has been calculated in the process of calculating the position of the next frame RGB image, it only needs to be directly obtained.

[0054] Another case is that, assuming that the target RGB image is not the first frame image in the preset initialization sliding window, that is, the corresponding depth image is not collected in the first frame RGB image in the sliding window. Then, when constructing the target 3D map of the current environment, because the RGB image was collected before the target RGB image, the map points can be restored not only based on the target RGB image, but also based on the RGB image collected before the target RGB image, so as to restore more map points to enrich the 3D map of the current environment.

[0055] At this time, the first pose transformation matrix between the next frame RGB image and the target RGB image is calculated in the same way. According to the first pose transformation matrix and the depth image, the initial three-dimensional map of the current environment can be constructed. Here, if the first pose transformation matrix has been calculated in the process of calculating the pose of the next frame RGB image, it only needs to be directly obtained. Then, the second pose transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated, and the initial three-dimensional map of the current environment is updated according to the second pose transformation matrix and the depth image to generate the target three-dimensional map.

[0056] Step 280, based on the target three-dimensional map, calculate the position and posture of the electronic device when collecting other RGB images located after the next frame of RGB image within the preset initialization sliding window.

[0057] After the target three-dimensional map is constructed, the position and posture of the electronic device when collecting other RGB images located after the next frame of RGB image in the sliding window can be calculated based on the target three-dimensional map. Since the target three-dimensional map is constructed based on the first frame of RGB image in the sliding window to the next frame of RGB image of the target RGB image, combined with the depth image of the target RGB image, that is, it is constructed based on the map points jointly restored by these RGB images and the depth image, it is obvious that the map points on the target three-dimensional map are more comprehensive than the depth image.

[0058] Therefore, by calculating the pose of the electronic device when collecting other RGB images located after the next frame of RGB image within the preset initialization sliding window, the pose of the other RGB images can be directly calculated according to the target three-dimensional map.

[0059] In an embodiment of the present application, a target three-dimensional map of the current environment is constructed based on the posture and depth image when the electronic device collects the next frame of RGB image. Since the target three-dimensional map is constructed based on the first frame of RGB image in the sliding window until the next frame of RGB image of the target RGB image, combined with the depth image of the target RGB image, it is obvious that the map points on the target three-dimensional map are more comprehensive relative to the depth image. Therefore, according to the target three-dimensional map, the posture of the electronic device when collecting other RGB images located after the next frame of RGB image in the preset initialization sliding window is calculated. The accuracy of the posture of other RGB images after the next frame of RGB image is greatly improved.

[0060] In one embodiment, step 260, constructing a target three-dimensional map of the current environment according to the position and depth image when the electronic device collects the next frame of RGB image, includes:

[0061] Calculate the first pose transformation matrix between the next frame RGB image and the target RGB image, and construct the target 3D map of the current environment based on the first pose transformation matrix and the depth image.

[0062] In one embodiment, if the target RGB image is the first frame image in the preset initialization sliding window, there is no RGB image before the target RGB image, so only the first pose transformation matrix between the next frame RGB image and the target RGB image needs to be calculated. Then, the target three-dimensional map of the current environment is constructed based on the first pose transformation matrix and the depth image.

[0063] In another embodiment, if the target RGB image is not the first frame image in the preset initialization sliding window, there is an RGB image before the target RGB image. Therefore, first, the first pose transformation matrix between the next frame RGB image and the target RGB image is calculated, and the initial three-dimensional map of the current environment is constructed according to the first pose transformation matrix and the depth image. Secondly, the second pose transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated, and the initial three-dimensional map of the current environment is updated according to the second pose transformation matrix and the depth image to generate a target three-dimensional map.

[0064] In an embodiment of the present application, if the target RGB image is the first frame image in the preset initialization sliding window, it is only necessary to calculate the first position transformation matrix between the next frame RGB image and the target RGB image. Then, according to the first position transformation matrix and the depth image, the target three-dimensional map of the current environment is directly constructed. If the target RGB image is not the first frame image in the preset initialization sliding window, the first position transformation matrix between the next frame RGB image and the target RGB image is first calculated. Then, according to the first position transformation matrix and the depth image, the initial three-dimensional map of the current environment is constructed. Finally, the second position transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated, and the initial three-dimensional map of the current environment is updated according to the second position transformation matrix and the depth image to generate the target three-dimensional map. Since the target three-dimensional map is constructed based on the first frame RGB image in the sliding window until the next frame RGB image of the target RGB image, combined with the depth image of the target RGB image, it is obvious that the map points on the target three-dimensional map are more comprehensive than the depth image. Therefore, according to the target three-dimensional map, the position and posture of the electronic device when collecting other RGB images after the next frame RGB image in the preset initialization sliding window are calculated. This greatly improves the pose accuracy of other RGB images calculated after the next frame of RGB image.

[0065] In one embodiment, constructing a target three-dimensional map of the current environment according to the position and depth image when the electronic device collects the next frame of RGB image, includes:

[0066] Construct an initial 3D map of the current environment based on the first pose transformation matrix and the depth image;

[0067] A second pose transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated, and an initial three-dimensional map of the current environment is updated according to the second pose transformation matrix and the depth image to generate a target three-dimensional map.

[0068] like Figure 4 As shown in FIG. 1 , a schematic diagram of constructing a target three-dimensional map of the current environment in an embodiment if the target RGB image is not the first frame image in the preset initialization sliding window. For example, assuming that there are 10 frames of images in the preset initialization sliding window, and a depth image is acquired at the 4th frame, that is, the target RGB image is the 4th frame RGB image. Then, when constructing the target three-dimensional map of the current environment at this time, there are specifically two steps:

[0069] The first step is to construct an initial three-dimensional map of the current environment. Specifically, the first pose transformation matrix between the next frame RGB image (the fifth frame) and the target RGB image (the fourth frame) is calculated, and the initial three-dimensional map of the current environment is constructed based on the first pose transformation matrix and the depth image. The method of calculating the initial three-dimensional map of the current environment here is the same as if the target RGB image is the first frame image in the preset initialization sliding window, then the method of constructing the target three-dimensional map of the current environment in step 260 is the same as the method of constructing the target three-dimensional map of the current environment based on the pose and depth image when the electronic device collects the next frame RGB image, so it will not be repeated here.

[0070] The second step is to update the initial 3D map to generate a target 3D map. Specifically, the second pose transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated, and the initial 3D map of the current environment is supplemented according to the second pose transformation matrix and the depth image to generate the target 3D map.

[0071] For the RGB image before the target RGB image, that is, for the 1st, 2nd, and 3rd frames, the second pose transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated, that is, the second pose transformation matrix between the 4th frame and the 1st, 2nd, and 3rd frames is calculated respectively. Then, the initial 3D map of the current environment is updated according to the second pose transformation matrix and the depth image to generate the target 3D map.

[0072] Specifically, first calculate the second pose transformation matrix between the 4th frame and the 1st frame, and then update the initial three-dimensional map of the current environment according to the second pose transformation matrix and the depth image to generate an updated three-dimensional map. Next, calculate the second pose transformation matrix between the 4th frame and the 2nd frame, and then update the updated three-dimensional map again according to the second pose transformation matrix and the depth image to generate a three-dimensional map that is updated again. Calculate the second pose transformation matrix between the 4th frame and the 3rd frame, and then update the three-dimensional map of the current environment that has been updated again according to the second pose transformation matrix and the depth image for the third time to generate a target three-dimensional map.

[0073] In an embodiment of the present application, if the target RGB image is not the first frame image within the preset initialization sliding window, when constructing the target three-dimensional map of the current environment, it includes: calculating the first pose transformation matrix between the next frame RGB image and the target RGB image, and constructing the initial three-dimensional map of the current environment based on the first pose transformation matrix and the depth image. Then calculate the second pose transformation matrix between the RGB image before the target RGB image and the target RGB image, and update the initial three-dimensional map of the current environment based on the second pose transformation matrix and the depth image to generate a target three-dimensional map. In the final target three-dimensional map, not only the map points corresponding to the next frame RGB image of the target RGB image are restored, but also the map points of each frame image before the target RGB image are restored. Therefore, the integrity and accuracy of the target three-dimensional map obtained at this time are greatly improved.

[0074] In one embodiment, a posture calculation method is provided, further comprising:

[0075] The preset perspective projection PnP algorithm is used to calculate the first pose transformation matrix or the second pose transformation matrix. The preset perspective projection PnP algorithm is used to calculate the rotation transformation matrix and the translation transformation matrix between the relevant RGB image of the target RGB image and the target RGB image in steps, and the relevant RGB image of the target RGB image is the RGB image before the target RGB image or the next frame RGB image.

[0076] Among them, calculating the first pose transformation matrix between the next frame RGB image and the target RGB image includes:

[0077] The preset perspective projection PnP algorithm is used to calculate the first pose transformation matrix (also referred to as pose) between the next frame RGB image and the target RGB image; wherein the preset perspective projection PnP algorithm is used to calculate the rotation transformation matrix and the translation transformation matrix between the next frame RGB image and the target RGB image step by step.

[0078] The preset perspective projection PnP algorithm is relative to the traditional perspective projection PnP algorithm. The traditional perspective projection PnP algorithm is based on 3D-2D feature point pairs to simultaneously calculate the rotation transformation matrix and translation transformation matrix between two frames. The preset perspective projection PnP algorithm is used to calculate the rotation transformation matrix and translation transformation matrix between two frames in steps.

[0079] Specifically, when the preset perspective projection PnP algorithm is used to calculate the first pose transformation matrix between the next frame RGB image and the target RGB image, first, the first rotation transformation matrix between the next frame RGB image and the target RGB image is calculated; second, the first translation transformation matrix between the next frame RGB image and the target RGB image is calculated. The first rotation transformation matrix and the first translation transformation matrix constitute the first pose transformation matrix.

[0080] Similarly, the first pose transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated, including:

[0081] A preset perspective projection PnP algorithm is used to calculate a second posture transformation matrix (also referred to as posture) between an RGB image located before a target RGB image and the target RGB image; wherein the preset perspective projection PnP algorithm is used to calculate a second rotation transformation matrix and a second translation transformation matrix between an RGB image located before a target RGB image and the target RGB image in steps. The second rotation transformation matrix and the second translation transformation matrix constitute a second posture transformation matrix.

[0082] In an embodiment of the present application, when calculating the posture transformation matrix between the RGB image before the target RGB image or the next frame RGB image and the target RGB image, the process of calculating the rotation transformation matrix and the translation transformation matrix is ​​decoupled. This can avoid superposition of the error generated when calculating the rotation transformation matrix and the error generated when calculating the translation transformation matrix. Thus, the accuracy of the first or second posture transformation matrix finally obtained by using the preset perspective projection PnP algorithm to achieve step-by-step calculation is improved.

[0083] In one embodiment, a preset perspective projection PnP algorithm is used to calculate the first pose transformation matrix or the second pose transformation matrix, including:

[0084] Calculate the rotation transformation matrix between the related RGB image of the target RGB image and the target RGB image;

[0085] According to the depth image and the rotation transformation matrix, the translation transformation matrix between the related RGB image of the target RGB image and the target RGB image is calculated;

[0086] Based on the rotation transformation matrix and the translation transformation matrix, a first pose transformation matrix or a second pose transformation matrix between the related RGB image of the target RGB image and the target RGB image is generated.

[0087] Among them, if the related RGB image of the target RGB image is the next frame RGB image, that is, there is no RGB image before the target RGB image, then Figure 5As shown, using the preset perspective projection PnP algorithm, only the first pose transformation matrix needs to be calculated, including:

[0088] Step 520, calculating a first rotation transformation matrix between the next frame RGB image and the target RGB image.

[0089] Since some areas in the collected depth image may have no depth information or inaccurate depth information, the 2D-2D feature point pairs that match between two RGB images are generally more numerous and comprehensive than the 3D-2D feature point pairs that match between one depth image and another RGB image.

[0090] Combination Figure 6 As shown, it is a schematic diagram of using a preset perspective projection PnP algorithm to calculate a posture transformation matrix in one embodiment. Figure 6 Image i is the target RGB image, image j is the next frame RGB image of the target RGB image, and the elliptical frame corresponds to the depth image corresponding to the target RGB image. Figure 6 As shown in the pair of images i and j on the left, the 2D-2D feature point pairs matching each other are determined in images i and j, and the first rotation transformation matrix R is calculated. ij Schematic diagram of .

[0091] Specifically, a pair of 2D-2D feature points that match each other can be determined between the next RGB image frame and the target RGB image. Specifically, an optical flow method or other image matching method can be used to determine the 2D-2D feature point pairs between the two frames. The first rotation transformation matrix R between the next RGB image frame and the target RGB image is calculated by combining the epipolar geometry constraint with the 2D-2D feature point pairs that match each other between the next RGB image frame and the target RGB image. ij .

[0092] Step 540: Calculate a first translation transformation matrix between the next frame RGB image and the target RGB image according to the depth image and the first rotation transformation matrix.

[0093] Specifically, first, the 2D-2D feature point pairs that match each other on the next frame RGB image and the target RGB image are removed according to the first rotation transformation matrix to obtain the removed 2D-2D feature point pairs. Secondly, the removed 2D-2D feature point pairs are converted into 3D-2D feature point pairs according to the depth image. Finally, the first translation transformation matrix t between the next frame RGB image and the target RGB image is calculated according to the 3D-2D feature point pairs. ij .

[0094] Step 560: Generate a first pose transformation matrix between the next frame RGB image and the target RGB image based on the first rotation transformation matrix and the first translation transformation matrix.

[0095] Combined with the first rotation transformation matrix R calculated above ij and the first translation transformation matrix t ij , generate the first pose transformation matrix between the next frame RGB image and the target RGB image.

[0096] In an embodiment of the present application, the first rotation transformation matrix between the next frame RGB image and the target RGB image is calculated. According to the depth image and the first rotation transformation matrix, the first translation transformation matrix between the next frame RGB image and the target RGB image is calculated. Based on the first rotation transformation matrix and the first translation transformation matrix, the first position transformation matrix between the next frame RGB image and the target RGB image is generated. The preset perspective projection PnP algorithm is used to realize the step-by-step calculation of the rotation transformation matrix and the translation transformation matrix, thereby avoiding the superposition of the errors generated in the calculation process of the two, thereby improving the accuracy of the first position transformation matrix finally obtained.

[0097] Continuing with the previous embodiment, Figure 7 As shown, step 540, calculating the first translation transformation matrix between the next frame RGB image and the target RGB image according to the depth image and the first rotation transformation matrix, includes:

[0098] Step 542: Eliminate the 2D-2D feature point pairs that match each other on the next frame RGB image and the target RGB image according to the first rotation transformation matrix to obtain the eliminated 2D-2D feature point pairs.

[0099] Combination Figure 6 As shown, image i is the target RGB image, image j is the next frame RGB image of the target RGB image, and the elliptical frame corresponds to the depth image corresponding to the target RGB image. Figure 6 As shown in the pair of images i and j on the right side, in images i and j, the 2D-2D feature point pairs that match each other on images i and j are eliminated according to the first rotation transformation matrix to obtain the eliminated 2D-2D feature point pairs.

[0100] Step 544 , converting the eliminated 2D-2D feature point pairs into 3D-2D feature point pairs according to the depth image.

[0101] Combination Figure 6The depth image shown in the oval frame in , converts the eliminated 2D-2D feature point pairs into 3D-2D feature point pairs according to the depth image. The Ransanc algorithm is used to eliminate the abnormal point pairs in the 3D-2D feature point pairs to generate the eliminated 3D-2D feature point pairs.

[0102] Step 546: Calculate a first translation transformation matrix between the next frame RGB image and the target RGB image based on the 3D-2D feature point pairs.

[0103] After calculating the 3D-2D feature point pairs between the target RGB image and the next frame RGB image, the first translation transformation matrix between the next frame RGB image and the target RGB image can be calculated based on the translation transformation matrix calculation formula.

[0104] The translation transformation matrix calculation formula is as follows:

[0105]

[0106] in, and is the 3D-2D matching point pair between image i and image j, R ij is the rotation transformation matrix from image j to image i, t ij is the translation transformation matrix from image j to image i, yes The corresponding depth value, π() -1 It is a transformation matrix that back-projects 2D points into 3D points.

[0107] For multiple 3D-2D feature point pairs, the least squares formula is constructed using the above formula (1-1) to calculate the optimal variable t ij , as the first translation transformation matrix between the next frame RGB image and the target RGB image.

[0108] In an embodiment of the present application, when calculating the first translation transformation matrix between the next frame RGB image and the target RGB image, first, the 2D-2D feature point pairs that match each other on the next frame RGB image and the target RGB image are eliminated according to the first rotation transformation matrix to obtain the eliminated 2D-2D feature point pairs. Secondly, the eliminated 2D-2D feature point pairs are converted into 3D-2D feature point pairs according to the depth image. Finally, based on the 3D-2D feature point pairs, the first translation transformation matrix between the next frame RGB image and the target RGB image is calculated. The feature point pairs are subjected to multiple elimination of abnormal point pairs, and the least squares method is used to calculate the first translation transformation matrix, thereby improving the accuracy of the calculated first translation transformation matrix.

[0109] In one embodiment, if the target RGB image is the first frame image in the preset initialization sliding window, the first position transformation matrix between the next frame RGB image and the target RGB image is calculated, such as Figure 8 As shown, according to the first pose transformation matrix and the depth image, a target three-dimensional map of the current environment is constructed, including:

[0110] Step 820: Project the 3D feature points on the depth image onto the next frame of RGB image according to the first pose transformation matrix to generate projected 2D feature points.

[0111] Among them, the first pose transformation matrix is ​​the pose transformation matrix between the next frame RGB image and the target RGB image. Based on the depth image and the target RGB image corresponding to the target RGB image, the 3D-2D matching point pairs can be determined, and then based on the first pose transformation matrix, the 3D feature points on the depth image can be projected onto the next frame RGB image to generate projected 2D feature points.

[0112] Step 840, calculating the reprojection error between the projected 2D feature points and the 2D feature points on the next frame of RGB image.

[0113] Step 860: If the reprojection error is less than a preset error threshold, the 3D feature point on the depth image is used as a target map point.

[0114] Specifically, there are many original 2D feature points on the next frame RGB image. At this time, the reprojection error between these projected 2D feature points and the original 2D feature point positions is calculated. If the reprojection error is less than the preset error threshold, the 3D feature point corresponding to the reprojection error less than the preset error threshold is considered to be a credible map point, and the 3D feature point on the depth image is used as the target map point.

[0115] Step 880, construct a target three-dimensional map of the current environment according to the target map points.

[0116] After the 3D feature points that meet the conditions in the 3D feature points on the depth image are used as target map points, a target three-dimensional map of the current environment can be constructed based on these target map points.

[0117] In an embodiment of the present application, if the target RGB image is the first frame image within a preset initialization sliding window, the 3D feature points on the depth image are projected onto the next frame RGB image according to the first pose transformation matrix to generate projected 2D feature points. The reprojection error between the projected 2D feature points and the 2D feature points on the next frame RGB image is calculated. If the reprojection error is less than the preset error threshold, the 3D feature points on the depth image are used as target map points, and a target three-dimensional map of the current environment is constructed based on the target map points. This achieves the calculation of the target three-dimensional map of the current environment when the target RGB image is the first frame image within the preset initialization sliding window. Then, when calculating the pose of the electronic device when collecting other RGB images located after the next frame RGB image within the preset initialization sliding window, the target three-dimensional map can be directly used for calculation.

[0118] In one embodiment, a preset perspective projection PnP algorithm is used to calculate the first pose transformation matrix or the second pose transformation matrix, including:

[0119] Calculate the rotation transformation matrix between the related RGB image of the target RGB image and the target RGB image;

[0120] According to the depth image and the rotation transformation matrix, the translation transformation matrix between the related RGB image of the target RGB image and the target RGB image is calculated;

[0121] Based on the rotation transformation matrix and the translation transformation matrix, a first pose transformation matrix or a second pose transformation matrix between the related RGB image of the target RGB image and the target RGB image is generated.

[0122] If the relevant RGB image of the target RGB image is the RGB image before the target RGB image and the next frame RGB image, that is, there is an RGB image before the target RGB image, then the preset perspective projection PnP algorithm is used, and the first pose transformation matrix and the second pose transformation matrix need to be calculated simultaneously. The process of calculating the first pose transformation matrix is ​​not repeated here, and the second pose transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated, including:

[0123] The preset perspective projection PnP algorithm is used to calculate the second pose transformation matrix between the RGB image before the target RGB image and the target RGB image.

[0124] The preset perspective projection PnP algorithm is relative to the traditional perspective projection PnP algorithm. The traditional perspective projection PnP algorithm is based on 3D-2D feature point pairs to simultaneously calculate the rotation transformation matrix and translation transformation matrix between two frames. The preset perspective projection PnP algorithm is used to calculate the rotation transformation matrix and translation transformation matrix between two frames in steps.

[0125] Specifically, when the preset perspective projection PnP algorithm is used to calculate the second posture transformation matrix between the RGB image before the target RGB image and the target RGB image, first, the second rotation transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated; secondly, the second translation transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated according to the depth image and the second rotation transformation matrix. The second rotation transformation matrix and the second translation transformation matrix constitute the second posture transformation matrix.

[0126] In an embodiment of the present application, when calculating the second posture transformation matrix between the RGB image before the target RGB image and the target RGB image, the process of calculating the second rotation transformation matrix and the second translation transformation matrix is ​​decoupled. This can avoid superposition of the error generated when calculating the second rotation transformation matrix and the error generated when calculating the second translation transformation matrix. Thus, the use of the preset perspective projection PnP algorithm is improved, and the second posture transformation matrix is ​​calculated step by step, thereby improving the accuracy of the second posture transformation matrix.

[0127] In one embodiment, calculating a second translation transformation matrix between an RGB image before the target RGB image and the target RGB image according to the depth image and the second rotation transformation matrix includes:

[0128] Step 1: According to the second rotation transformation matrix, the 2D-2D feature point pairs that match each other on the RGB image before the target RGB image and the target RGB image are eliminated to obtain the eliminated 2D-2D feature point pairs.

[0129] Combination Figure 6 As shown, assuming that image i is the target RGB image, image j is the RGB image before the target RGB image, and the elliptical frame corresponds to the depth image corresponding to the target RGB image. Figure 6 As shown in the pair of images i and j on the right side, in images i and j, the 2D-2D feature point pairs that match each other on images i and j are eliminated according to the second rotation transformation matrix to obtain the eliminated 2D-2D feature point pairs.

[0130] Step 2: Convert the eliminated 2D-2D feature point pairs into 3D-2D feature point pairs according to the depth image.

[0131] Combination Figure 6The depth image shown in the oval frame in , converts the eliminated 2D-2D feature point pairs into 3D-2D feature point pairs according to the depth image. The Ransanc algorithm is used to eliminate the abnormal point pairs in the 3D-2D feature point pairs to generate the eliminated 3D-2D feature point pairs.

[0132] Step three, calculating a second translation transformation matrix between the RGB image before the target RGB image and the target RGB image based on the 3D-2D feature point pairs.

[0133] After calculating the 3D-2D feature point pairs between the target RGB image and the RGB image before the target RGB image, the second translation transformation matrix between the RGB image before the target RGB image and the target RGB image can be calculated based on the translation transformation matrix calculation formula. The translation transformation matrix calculation formula is as follows:

[0134]

[0135] in, and is the 3D-2D matching point pair between image i and image j, R ij is the rotation transformation matrix from image j to image i, t ij is the translation transformation matrix from image j to image i, yes The corresponding depth value, π() -1 It is a transformation matrix that back-projects 2D points into 3D points.

[0136] For multiple 3D-2D feature point pairs, the least squares formula is constructed using the above formula (1-1) to calculate the optimal variable t ij , as the second translation transformation matrix between the RGB image before the target RGB image and the target RGB image.

[0137] In an embodiment of the present application, when calculating the second translation transformation matrix between the RGB image before the target RGB image and the target RGB image, first, the 2D-2D feature point pairs that match each other on the RGB image before the target RGB image and the target RGB image are eliminated according to the second rotation transformation matrix to obtain the eliminated 2D-2D feature point pairs. Secondly, the eliminated 2D-2D feature point pairs are converted into 3D-2D feature point pairs according to the depth image. Finally, based on the 3D-2D feature point pairs, the second translation transformation matrix between the RGB image before the target RGB image and the target RGB image is calculated. The feature point pairs are subjected to multiple elimination of abnormal point pairs, and the least squares method is used to calculate the second translation transformation matrix, thereby improving the accuracy of the calculated second translation transformation matrix.

[0138] In one embodiment, updating the initial three-dimensional map of the current environment according to the second posture transformation matrix and the depth image to generate a target three-dimensional map includes:

[0139] According to the second posture transformation matrix, the 3D feature points on the depth image are projected onto the RGB image located before the target RGB image to generate projected 2D feature points;

[0140] Calculate the reprojection error between the projected 2D feature points and the 2D feature points on the RGB image before the target RGB image;

[0141] If the reprojection error is less than the preset error threshold, the 3D feature point on the depth image is used as the new target map point;

[0142] Add new target map points to the initial three-dimensional map to generate a target three-dimensional map.

[0143] Specifically, the second pose transformation matrix is ​​the pose transformation matrix between the RGB image before the target RGB image and the target RGB image. Based on the depth image and the target RGB image corresponding to the target RGB image, the 3D-2D matching point pairs can be determined, and then based on the second pose transformation matrix, the 3D feature points on the depth image can be projected onto the RGB image before the target RGB image to generate projected 2D feature points.

[0144] There are many original 2D feature points on the RGB image before the target RGB image. At this time, the reprojection error between these projected 2D feature points and the original 2D feature point positions is calculated. If the reprojection error is less than the preset error threshold, the 3D feature point corresponding to the reprojection error less than the preset error threshold is considered to be a credible map point, and the 3D feature point is used as the target map point.

[0145] After the 3D feature points that meet the conditions in the 3D feature points on the depth image are used as target map points, a target three-dimensional map of the current environment can be constructed based on these target map points.

[0146] In an embodiment of the present application, if the target RGB image is not the first frame image within the preset initialization sliding window, the 3D feature points on the depth image are projected onto the RGB image before the target RGB image according to the second posture transformation matrix to generate projected 2D feature points. The reprojection error between the projected 2D feature points and the 2D feature points on the RGB image before the target RGB image is calculated. If the reprojection error is less than the preset error threshold, the 3D feature points on the depth image are used as target map points, and a target three-dimensional map of the current environment is constructed based on the target map points. This achieves the calculation of the target three-dimensional map of the current environment when the target RGB image is not the first frame image within the preset initialization sliding window. Then, when calculating the posture of the electronic device when collecting other RGB images located after the next frame RGB image within the preset initialization sliding window, the target three-dimensional map can be directly used for calculation.

[0147] In one embodiment, Fig. 9 As shown, step 280, according to the target three-dimensional map, calculates the position and posture of the electronic device when collecting the RGB image after the next frame of RGB image in the preset initialization sliding window, including:

[0148] Step 282, taking the next frame of the next RGB image as the current frame, and performing the following target operations:

[0149] Step 284, generating 3D-2D feature point pairs matching the target three-dimensional map and the current frame according to the target three-dimensional map, the current frame and the RGB image before the current frame;

[0150] Step 286, calculating the pose of the current frame based on the 3D-2D feature point pairs;

[0151] Step 288: Update the target three-dimensional map according to the current frame, and use the updated three-dimensional map as a new target three-dimensional map.

[0152] The next frame of the current frame is used as the new current frame, and the target operation is performed cyclically until the pose of the last frame of the RGB image in the preset initialization sliding window is calculated.

[0153] Combination Fig.10 As shown, it is a schematic diagram of calculating the pose and three-dimensional map calculation when the RGB image after the next RGB image is collected in one embodiment. For example, assuming that there are 10 frames of images in the preset initialization sliding window, the depth image is collected at the 4th frame, that is, the target RGB image is the 4th frame, and the next RGB image of the target RGB image is the 5th frame. At this time, the pose when the RGB image after the next RGB image is collected is calculated, that is, the pose when the 6th to 10th frames of images are collected.

[0154] Starting from calculating the pose when capturing the 6th frame of image, first, the next frame (6th frame) of the next RGB image (5th frame) is used as the current frame to calculate the pose of the current frame.

[0155] Secondly, according to the target three-dimensional map, the RGB image before the current frame and the current frame (the sixth frame), a 3D-2D feature point pair matching each other between the target three-dimensional map and the current frame (the sixth frame) is generated. Specifically, from the RGB image before the current frame and the current frame, an optical flow method or other image matching method is used to obtain a pair of 2D feature points that match each other. 3D feature points that match the 2D feature point pairs are obtained from the map points in the target three-dimensional map, and a 3D-2D feature point pair matching each other between the target three-dimensional map and the current frame is generated according to the matched 3D feature points and 2D feature point pairs.

[0156] Again, based on the 3D-2D feature point pairs, the traditional perspective projection PnP algorithm is used to calculate the pose of the current frame. Among them, the traditional perspective projection PnP algorithm simultaneously calculates the rotation transformation matrix and the translation transformation matrix between the two frames based on the 3D-2D feature point pairs. Therefore, when the traditional perspective projection PnP algorithm is used to calculate the pose of the current frame, the rotation transformation matrix and the translation transformation matrix between the RGB image before the current frame and the current frame are calculated based on the 3D-2D feature point pairs. Furthermore, based on the rotation transformation matrix and the translation transformation matrix, the pose transformation matrix is ​​obtained, and then the pose of the current frame is obtained based on the pose transformation matrix and the pose of the RGB image before the current frame. Specifically, the pose transformation matrix can be obtained based on the multiplication of the rotation transformation matrix and the translation transformation matrix, and this application does not limit this calculation method.

[0157] Finally, the target 3D map is updated according to the current frame, and the updated 3D map is used as the new target 3D map. The next RGB image of the current frame (the 7th frame) is used as the new current frame, and the target operation steps 284-286 are executed repeatedly until the pose of the last RGB image in the preset initialization sliding window is calculated.

[0158] In an embodiment of the present application, in the process of calculating the pose of the electronic device when collecting the RGB image after the next RGB image frame in the preset initialization sliding window, a loop method is adopted. First, based on the target three-dimensional map and the current frame and the RGB image before the current frame, the pose of the current frame is calculated. Secondly, the target three-dimensional map is updated based on the current frame, and the updated three-dimensional map is used as the new target three-dimensional map. Finally, based on the new target three-dimensional map and the next set of current frames and the RGB images before the current frame, the pose of the new current frame is calculated. Secondly, based on the new current frame, the new target three-dimensional map is updated again, and the three-dimensional map updated again is used as the new target three-dimensional map. This cycle is repeated until the pose of the last RGB image frame in the preset initialization sliding window is calculated.

[0159] Since the three-dimensional map calculated based on all the previous frames is used when calculating the position and posture of the subsequent frame, the accuracy of the calculated position and posture of the subsequent frame is improved.

[0160] In one embodiment, generating a 3D-2D feature point pair matching the target 3D map and the current frame according to the target 3D map, the current frame and the RGB image before the current frame includes:

[0161] Obtain mutually matching 2D feature point pairs from the current frame and the RGB image before the current frame;

[0162] 3D feature points matching the 2D feature point pairs are obtained from the map points in the target 3D map, and 3D-2D feature point pairs matching the target 3D map and the current frame are generated based on the matched 3D feature points and 2D feature point pairs.

[0163] In an embodiment of the present application, an optical flow method or other image matching method is used to obtain mutually matching 2D feature point pairs from the current frame and the RGB image before the current frame. Because the RGB images before the current frame provide some map points when calculating the target three-dimensional map, 3D feature points that match the 2D feature point pairs can be obtained from the map points in the target three-dimensional map. Then, based on the 3D feature points and the 2D feature points of the current frame in the 2D feature point pairs, 3D-2D feature point pairs that match the target three-dimensional map and the current frame are generated. Therefore, based on the 3D-2D feature point pairs, the matching relationship between the 3D feature points on the target three-dimensional map and the 2D feature points on the current frame is obtained.

[0164] In one embodiment, the current frame is an image frame after the next frame of the target RGB image in the sliding window. If the current frame has a corresponding depth image, step 288 updates the target three-dimensional map according to the current frame to generate an updated three-dimensional map, including:

[0165] According to the depth image corresponding to the current frame and the pose transformation matrix between the current frame and the RGB image before the current frame in the preset sliding window, the target three-dimensional map is updated to generate an intermediate three-dimensional map;

[0166] The intermediate three-dimensional map is updated by using a triangulation method to generate an updated three-dimensional map.

[0167] Assuming that there is a corresponding depth image for the current frame (the 6th frame), when the target 3D map is updated according to the current frame to generate an updated 3D map, the target 3D map is first updated to generate an intermediate 3D map based on the depth image corresponding to the current frame and the pose transformation matrix between the current frame and the RGB image before the current frame in the preset sliding window.

[0168] Specifically, first, calculate the pose transformation matrix between the current frame and the RGB image before the current frame in the preset sliding window; secondly, project the 3D feature points on the depth image of the current frame onto the RGB image before the current frame according to the pose transformation matrix to generate projected 2D feature points. Thirdly, calculate the reprojection error between the projected 2D feature points and the 2D feature points on the RGB image before the current frame. If the reprojection error is less than the preset error threshold, the 3D feature points on the depth image of the current frame are used as the target map points. Finally, update the target map according to the target map points to generate an intermediate three-dimensional map.

[0169] Then, the intermediate 3D map is updated by triangulation to generate an updated 3D map. In multi-view geometry, the camera observes the same spatial point at two locations, and the coordinates of the 3D spatial point are obtained by the two camera poses and the image observation point coordinates. This process is the calculation process of the triangulation method. The triangulation method can restore the depth information missing in some depth images.

[0170] In an embodiment of the present application, if there is a corresponding depth image for the current frame, the target three-dimensional map is updated to generate an intermediate three-dimensional map based on the depth image corresponding to the current frame and the pose transformation matrix between the current frame and the RGB image before the current frame in the preset sliding window. The intermediate three-dimensional map is then updated using the triangulation method to generate an updated three-dimensional map. If there is a corresponding depth image for the current frame, then the target three-dimensional map is updated based on the 3D feature points on the depth image of the current frame in combination with the target three-dimensional map constructed by the previous image frame. Finally, the triangulation method can be used to restore the depth information missing in some depth images. Thereby, the integrity and accuracy of the three-dimensional map constructed at this time are greatly improved.

[0171] Continuing from the previous embodiment, if there is no corresponding depth image for the current frame, step 288 is to update the target three-dimensional map according to the current frame to generate an updated three-dimensional map, including:

[0172] The target three-dimensional map is updated by using a triangulation method to generate an updated three-dimensional map.

[0173] In the embodiment of the present application, if there is no corresponding depth image in the current frame, the target three-dimensional map is updated by triangulation to generate an updated three-dimensional map. Since the triangulation method can restore the missing depth information in some depth images, the integrity and accuracy of the three-dimensional map constructed at this time are greatly improved to a certain extent.

[0174] In one embodiment, Fig.11 As shown, after calculating the position and posture of the electronic device when collecting the RGB image after the next frame of RGB image within the preset initialization sliding window according to the target three-dimensional map, it also includes:

[0175] Step 1120, obtaining the IMU data collected within a preset sliding window.

[0176] The VIO (Visual-Inertial Odometry) system can be used to fuse the image data collected by the camera with the IMU (Inertial measurement unit) data to achieve more accurate positioning.

[0177] Since the frequency of IMU data acquisition in electronic devices is greater than the frequency of RGB image acquisition, when the sliding window contains 10 frames of RGB images, the collected IMU data is generally more than 10 groups. When calculating the initialization information of the IMU, the calculation is based on all the IMU data collected in the preset sliding window. Therefore, it is necessary to obtain all the IMU data collected in the preset sliding window.

[0178] Step 1140, calculate the initialization information of the IMU according to the position and posture of each frame of the RGB image in the preset sliding window and the IMU data; the initialization information includes the initial velocity, the zero bias of the IMU and the gravity vector of the IMU.

[0179] Then, according to the posture of the RGB image and all the IMU data collected in the preset sliding window, the rotation transformation matrix in the posture transformation matrix of the RGB image is used for rotation constraint, and the translation transformation matrix of the RGB image is used for translation constraint, so as to calculate the initialization information of the IMU. The initialization information of the IMU includes the initial speed of the electronic device, the zero bias of the IMU, and the gravity vector of the IMU.

[0180] Step 1160, calculating the pose of the RGB image collected after the preset sliding window according to the initial pose, the target three-dimensional map and the initialization information of the IMU.

[0181] After calculating the initialization information of the IMU, the pose of the RGB image collected after the preset sliding window can be calculated based on the initial pose, the target three-dimensional map and the initialization information of the IMU. For example, the pose of the 11th frame image is calculated. Of course, this application does not limit this. Among them, the initial pose is the pose of the target RGB image corresponding to the depth image of the current environment collected for the first time. At this time, the target three-dimensional map is a three-dimensional map constructed based on all image frames in the sliding window.

[0182] In the embodiment of the present application, when calculating the pose of the RGB image collected after the preset sliding window, the IMU is first initialized, and then the pose of the RGB image collected after the preset sliding window can be calculated in combination with not only the image data collected by the camera, but also the IMU data. The adaptability and robustness of pose calculation are improved from the two dimensions of vision and IMU.

[0183] In one embodiment, a posture calculation method is provided, further comprising:

[0184] If the depth image of the current environment is not collected within the preset initialization sliding window, the initial pose and initial 3D map are calculated based on the RGB image collected within the preset initialization sliding window;

[0185] According to the initial posture and the initial three-dimensional map, the posture of the electronic device when collecting the RGB image after the preset initialization sliding window is calculated.

[0186] In an embodiment of the present application, if a depth image of the current environment is not collected within the preset initialization sliding window, the traditional VINS-MONO algorithm is used to calculate the initial posture and the initial three-dimensional map based on the RGB image collected within the preset initialization sliding window. Then, based on the initial posture and the initial three-dimensional map, the posture of the electronic device when collecting the RGB image after the preset initialization sliding window is calculated. This ensures that when a depth image of the current environment is not collected within the preset initialization sliding window, the posture of the electronic device when collecting the RGB image can still be output in real time after accumulating 10 frames of RGB images within the sliding window.

[0187] In one embodiment, a posture calculation method is provided, further comprising:

[0188] If the target RGB image corresponding to the depth image is the last frame image in the preset initialization sliding window, the initial pose and the initial three-dimensional map are calculated according to the RGB image collected in the preset initialization sliding window;

[0189] According to the initial posture and the initial three-dimensional map, the posture of the electronic device when collecting the RGB image after the preset sliding window is calculated.

[0190] Combination Fig.12 , which is a schematic diagram of calculating the pose when the depth image of the current environment is not collected within the preset initialization sliding window, or the target RGB image corresponding to the depth image is the last frame image within the preset initialization sliding window.

[0191] In an embodiment of the present application, if the target RGB image corresponding to the depth image is the last frame image in the preset initialization sliding window, then if the posture of the electronic device when collecting the depth image is determined as the initial posture in the present application. According to the initial posture, the target RGB image, the depth image and the next frame RGB image of the target RGB image, the posture of the electronic device when collecting the next frame RGB image is determined. Therefore, the posture calculation method in the present application can only calculate the posture of the electronic device when collecting the RGB image after the sliding window, which is basically the same as the duration of the posture of the electronic device when collecting the RGB image after the preset initialization sliding window using the traditional VINS-MONO algorithm. Therefore, if the target RGB image corresponding to the depth image is the last frame image in the preset initialization sliding window, the traditional VINS-MONO algorithm can be used, or the posture calculation method in the present application can be used for calculation. Therefore, a variety of posture calculation methods are provided, which is more flexible.

[0192] In a specific embodiment, Fig.13 As shown, a posture calculation method is provided, and an RGB image of a depth image acquired for the first time is used as the second frame image in a preset initialization sliding window and the image after the second frame image as an example for explanation, and the method includes:

[0193] Step 1302, VIO system starts initialization;

[0194] Step 1304, collecting RGB images, IMU data and depth images;

[0195] Step 1306, performing dedistortion and alignment processing on the collected RGB image and depth image;

[0196] Step 1308, counting the frame number of the collected RGB image as frame_count;

[0197] Step 1310, determining whether the currently acquired RGB image has a corresponding depth image (Depth map), and whether it is the first Depth map acquired for the current environment; if so, proceeding to step 1312; if not, proceeding to step 1318;

[0198] Step 1312, set the flag start_depth_init=true for using the depth map to calculate the pose;

[0199] Step 1314, setting the image coordinate system of the RGB image to the world coordinate system, taking the pose of the RGB image as the initial pose, and setting the initial pose to 0; and recording the frame_count of the RGB image as first_depth;

[0200] Step 1316, when the next frame of RGB image is acquired, frame_count is updated to frame_count+1;

[0201] Step 1318, determine whether first_depth is less than the sliding window size windowsize (10 frames); if so, proceed to step 1320; if not, proceed to step 1354;

[0202] Step 1320, determine whether the frame number frame_count+1 of the current frame is equal to first_depth+1; if so, proceed to step 1322; if not, proceed to step 1334;

[0203] Step 1322, using a preset perspective projection PnP algorithm, calculate the first pose between the first_depth frame and the first_depth+1 frame;

[0204] Step 1324, projecting the 3D feature points on the Depth map onto the first_depth+1 frame according to the first pose to generate projected 2D feature points; calculating the reprojection error between the projected 2D feature points and the 2D feature points on the first_depth+1 frame;

[0205] Step 1326: If the reprojection error is less than a preset error threshold, the 3D feature points on the Depth map are used as target map points; and an initial three-dimensional map of the current environment is constructed based on the target map points.

[0206] Step 1328, calculate a second pose transformation matrix between the RGB image before the first_depth frame and the first_depth frame;

[0207] Step 1330, projecting the 3D feature points on the Depth image onto the RGB image before the first_depth frame according to the second pose transformation matrix to generate projected 2D feature points; calculating the reprojection error between the projected 2D feature points and the 2D feature points on the RGB image before the first_depth frame;

[0208] Step 1332: if the reprojection error is less than a preset error threshold, the 3D feature point on the Depth map is used as a new target map point; the new target map point is added to the initial three-dimensional map to generate a target three-dimensional map;

[0209] Step 1334, generating 3D-2D feature point pairs matching the target 3D map and the current frame according to the target 3D map, the current frame and the RGB image before the current frame;

[0210] Step 1336, based on the 3D-2D feature point pairs, the conventional PnP algorithm is used to calculate the pose of the current frame; and the process proceeds to step 1354, outputting the pose;

[0211] Step 1338, determine whether there is a corresponding Depth map for the current frame; if so, proceed to step 1340; if not, proceed to step 1344;

[0212] Step 1340, updating the target three-dimensional map to generate an intermediate three-dimensional map according to the depth image corresponding to the current frame and the pose transformation matrix between the current frame and the RGB image before the current frame in the preset sliding window;

[0213] Step 1342, using a triangulation method to update the intermediate three-dimensional map to generate an updated three-dimensional map;

[0214] Step 1344, updating the target three-dimensional map using a triangulation method to generate an updated three-dimensional map;

[0215] Step 1346, determine whether frame_count is equal to windowsize (10 frames); if so, proceed to step 1348;

[0216] Step 1348, performing BA optimization on the calculated poses corresponding to the 10 frames of images;

[0217] Step 1350, initializing the IMU based on the poses corresponding to the 10 frames of images after BA optimization;

[0218] Step 1352, calculate the pose of the RGB image collected after the preset sliding window according to the initial pose, the target three-dimensional map and the initialization information of the IMU.

[0219] Step 1354, using the VINS-MONO algorithm to calculate the pose.

[0220] In an embodiment of the present application, it is not necessary to collect a full RGB image within the preset initialization sliding window before the electronic device can output the posture when collecting the first frame of RGB image outside the preset initialization sliding window. Instead, when the depth image of the current environment is first collected within the preset initialization sliding window, and the target RGB image corresponding to the depth image is not the last frame of image within the preset initialization sliding window, the posture of the electronic device when collecting the depth image can be determined as the initial posture, target RGB image, depth image and the next frame of RGB image of the target RGB image, and the posture of the electronic device when collecting the next frame of RGB image is determined. The time for the first output of the posture is greatly advanced from outside the preset initialization sliding window to inside the sliding window and to the next frame when the depth image is first received. Thus, the posture can be output in real time within the preset initialization sliding window, reducing the waiting time for real-time output of the posture.

[0221] If the depth image of the current environment is not collected within the preset initialization sliding window or the target RGB image corresponding to the depth image is the last frame of the image within the preset initialization sliding window, the traditional VINS-MONO algorithm is used to calculate the position and posture of the electronic device when collecting the RGB image after the preset initialization sliding window based on the RGB image collected within the preset initialization sliding window. This ensures that if the depth image of the current environment is not collected within the preset initialization sliding window, the position and posture of the electronic device when collecting the RGB image can still be output in real time after accumulating 10 frames of RGB images within the sliding window.

[0222] At the same time, it is not necessary to collect a depth image corresponding to each frame of RGB image before the pose can be output in real time. Instead, a frequency lower than the frequency of collecting RGB images can be used to collect depth maps, or a variable frequency can be used to collect depth maps, and as long as the depth map is collected within the sliding window, initialization can be started, and the pose can be output in real time after initialization. Since the depth map can be collected at a lower frequency or a variable frequency, the pose can be initialized and output in real time based on the low-frequency depth map, thereby avoiding the collection of a large amount of data and the processing of a large amount of data, thereby reducing the power consumption of electronic devices.

[0223] In one embodiment, Fig.14 As shown, a posture calculation device 1400 is provided, and the device includes:

[0224] The initial posture determination module 1420 is used to determine the posture of the electronic device when the depth image is collected as the initial posture if the depth image of the current environment is collected for the first time within the preset initialization sliding window; wherein the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window;

[0225] The next frame RGB image posture determination module 1440 is used to determine the posture of the electronic device when capturing the next frame RGB image based on the initial posture, the target RGB image, the depth image and the next frame RGB image of the target RGB image.

[0226] In one embodiment, Fig.15 As shown, a posture calculation device 1400 is provided, and the device also includes:

[0227] The target three-dimensional map construction module 1460 is used to construct a target three-dimensional map of the current environment according to the posture and depth image when the electronic device collects the next frame of RGB image;

[0228] The other RGB image posture determination module 1480 is used to calculate the posture of the electronic device when collecting other RGB images located after the next frame of RGB image within a preset initialization sliding window according to the target three-dimensional map.

[0229] In one embodiment, the target three-dimensional map construction module 1460 further includes:

[0230] An initial three-dimensional map construction unit, used to construct an initial three-dimensional map of the current environment according to the first pose transformation matrix and the depth image;

[0231] The target three-dimensional map construction unit is used to calculate a second posture transformation matrix between an RGB image located before the target RGB image and the target RGB image, and to update the initial three-dimensional map of the current environment according to the second posture transformation matrix and the depth image to construct a target three-dimensional map of the current environment.

[0232] In one embodiment, a posture calculation device 1400 is provided, and the device further includes:

[0233] The posture transformation matrix calculation unit is further used to calculate the first posture transformation matrix or the second posture transformation matrix by using a preset perspective projection PnP algorithm;

[0234] Among them, the preset perspective projection PnP algorithm is used to step by step calculate the rotation transformation matrix and the translation transformation matrix between the related RGB image of the target RGB image and the target RGB image, and the related RGB image of the target RGB image is the RGB image before the target RGB image or the next frame RGB image.

[0235] In one embodiment, the posture transformation matrix calculation unit is further used to further include:

[0236] A rotation transformation matrix calculation subunit, used to calculate a rotation transformation matrix between a related RGB image of a target RGB image and the target RGB image;

[0237] A translation transformation matrix calculation subunit, used to calculate the translation transformation matrix between the related RGB image of the target RGB image and the target RGB image according to the depth image and the rotation transformation matrix;

[0238] The posture transformation matrix calculation subunit is used to generate a first posture transformation matrix or a second posture transformation matrix between the related RGB image of the target RGB image and the target RGB image based on the rotation transformation matrix and the translation transformation matrix.

[0239] In one embodiment, the translation transformation matrix calculation subunit is further used to eliminate the 2D-2D feature point pairs that match each other on the relevant RGB image of the target RGB image and the target RGB image according to the rotation transformation matrix to obtain the eliminated 2D-2D feature point pairs; convert the eliminated 2D-2D feature point pairs into 3D-2D feature point pairs according to the depth image; and calculate the translation transformation matrix between the relevant RGB image of the target RGB image and the target RGB image according to the 3D-2D feature point pairs.

[0240] In one embodiment, the initial three-dimensional map construction unit is also used to project the 3D feature points on the depth image onto the next frame RGB image according to the first pose transformation matrix to generate projected 2D feature points; calculate the reprojection error between the projected 2D feature points and the 2D feature points on the next frame RGB image; if the reprojection error is less than a preset error threshold, use the 3D feature points on the depth image as target map points; and construct an initial three-dimensional map of the current environment based on the target map points.

[0241] In one embodiment, the target three-dimensional map construction unit is also used to project the 3D feature points on the depth image onto the RGB image located before the target RGB image according to the second pose transformation matrix to generate projected 2D feature points; calculate the reprojection error between the projected 2D feature points and the 2D feature points on the RGB image located before the target RGB image; if the reprojection error is less than a preset error threshold, use the 3D feature points on the depth image as new target map points; add the new target map points to the initial three-dimensional map to construct a target three-dimensional map of the current environment.

[0242] In one embodiment, the other RGB image pose determination module 1480 includes:

[0243] The current frame definition unit is used to take the next frame of the next RGB image as the current frame and perform the following target operations:

[0244] The target operation unit is used to generate a 3D-2D feature point pair matching the target 3D map and the current frame according to the target 3D map, the current frame and the RGB image before the current frame; calculate the pose of the current frame based on the 3D-2D feature point pair; update the target 3D map according to the current frame, and use the updated 3D map as a new target 3D map;

[0245] The loop unit is used to take the next frame of the current frame as the new current frame and cyclically perform the target operation until the pose of the last frame of the RGB image in the preset initialization sliding window is calculated.

[0246] In one embodiment, the target operation unit is further used to obtain mutually matching 2D feature point pairs from the current frame and the RGB image before the current frame; obtain 3D feature points matching the 2D feature point pairs from the map points in the target three-dimensional map, and generate mutually matching 3D-2D feature point pairs between the target three-dimensional map and the current frame based on the matching 3D feature points and 2D feature point pairs.

[0247] In one embodiment, if there is a corresponding depth image for the current frame, the target operation unit is also used to update the target three-dimensional map to generate an intermediate three-dimensional map based on the depth image corresponding to the current frame and the posture transformation matrix between the current frame and the RGB image before the current frame in a preset sliding window; and use a triangulation method to update the intermediate three-dimensional map to generate an updated three-dimensional map.

[0248] In one embodiment, if there is no corresponding depth image for the current frame, the target operation unit is further configured to update the target three-dimensional map using a triangulation method to generate an updated three-dimensional map.

[0249] In one embodiment, Fig.16 As shown, a posture calculation device 1600 is provided, and the device also includes:

[0250] An IMU data acquisition module 1620 is used to acquire IMU data collected within a preset sliding window;

[0251] The IMU initialization module 1640 is used to calculate the initialization information of the IMU according to the position and posture of each frame of the RGB image in the preset sliding window and the IMU data; the initialization information includes the initial velocity, the zero bias of the IMU and the gravity vector of the IMU;

[0252] The first pose calculation module 1660 is used to calculate the pose of the RGB image collected after the preset sliding window according to the initial pose, the target three-dimensional map and the initialization information of the IMU.

[0253] In one embodiment, a posture calculation device is provided, the device further comprising:

[0254] A second posture calculation module is used to calculate an initial posture and an initial three-dimensional map according to the RGB image collected in the preset initialization sliding window if the depth image of the current environment is not collected in the preset initialization sliding window;

[0255] According to the initial posture and the initial three-dimensional map, the posture of the electronic device when collecting the RGB image after the preset initialization sliding window is calculated.

[0256] In one embodiment, a posture calculation device is provided, the device further comprising:

[0257] A third posture calculation module is used to calculate an initial posture and an initial three-dimensional map according to the RGB image collected in the preset initialization sliding window if the target RGB image corresponding to the depth image is the last frame image in the preset initialization sliding window;

[0258] According to the initial posture and the initial three-dimensional map, the posture of the electronic device when collecting the RGB image after the preset sliding window is calculated.

[0259] It should be understood that, although the various steps in the flowchart in the figure are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0260] The division of each module in the posture calculation device is only for illustration. In other embodiments, the posture calculation device can be divided into different modules as needed to complete all or part of the functions of the posture calculation device.

[0261] For the specific definition of the posture calculation device, please refer to the definition of the posture calculation method above, which will not be repeated here. Each module in the posture calculation device can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0262] In one embodiment, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of a posture calculation method provided in the above embodiments.

[0263] Fig.17 FIG. 1 is a schematic diagram of the internal structure of an electronic device in one embodiment. Fig.17 As shown, the electronic device includes a processor and a memory connected via a system bus. The processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement a posture calculation method provided in each of the above embodiments. The internal memory provides a cached operating environment for the operating system computer program in the non-volatile storage medium. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a vehicle-mounted computer, a wearable device, etc.

[0264] The implementation of each module in the posture calculation device provided in the embodiment of the present application can be in the form of a computer program. The computer program can be run on an electronic device or an electronic device. The program module constituted by the computer program can be stored in the memory of the electronic device or the electronic device. When the computer program is executed by the processor, the steps of the method described in the embodiment of the present application are implemented.

[0265] The embodiment of the present application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the posture calculation method.

[0266] A computer program product comprising instructions, when executed on a computer, causes the computer to perform a pose calculation method.

[0267] Any reference to memory, storage, database or other media used in the embodiments of the present application may include non-volatile and / or volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0268] The above initialization embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A posture calculation method, characterized in that: The method comprises: If a depth image of the current environment is collected for the first time within a preset initialization sliding window, the posture of the electronic device when collecting the depth image is determined as the initial posture; wherein the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window; and the acquisition frequency of collecting the depth image is lower than the acquisition frequency of collecting the target RGB image; According to the target RGB image, the depth image and the next frame RGB image of the RGB image, determine the posture transformation matrix between the target RGB image and the next frame RGB image of the RGB image; according to the initial posture and the posture transformation matrix, determine the posture of the electronic device when capturing the next frame RGB image.

2. The method for calculating the posture according to claim 1, characterized in that: The method further comprises: Calculate the first pose transformation matrix between the next frame RGB image and the target RGB image, and construct a target three-dimensional map of the current environment according to the first pose transformation matrix and the depth image; The next frame of the next RGB image is taken as the current frame, and a 3D-2D feature point pair that matches the target three-dimensional map and the current frame is generated according to the target three-dimensional map, the current frame and the RGB image before the current frame. Based on the 3D-2D feature point pair, the posture of the electronic device when collecting other RGB images after the next RGB image within the preset initialization sliding window is calculated.

3. The method for calculating the posture according to claim 2, characterized in that: The step of constructing a target three-dimensional map of the current environment according to the first posture transformation matrix and the depth image includes: Constructing an initial three-dimensional map of the current environment according to the first posture transformation matrix and the depth image; A second pose transformation matrix between an RGB image preceding the target RGB image and the target RGB image is calculated, and an initial three-dimensional map of the current environment is updated according to the second pose transformation matrix and the depth image to construct a target three-dimensional map of the current environment.

4. The method for calculating the posture according to claim 3, characterized in that: The method further comprises: Using a preset perspective projection PnP algorithm to calculate the first posture transformation matrix or the second posture transformation matrix; Among them, the preset perspective projection PnP algorithm is used to step by step calculate the rotation transformation matrix and the translation transformation matrix between the related RGB image of the target RGB image and the target RGB image, and the related RGB image of the target RGB image is the RGB image located before the target RGB image or the next frame RGB image.

5. The method for calculating the position and posture according to claim 4, characterized in that: The using a preset perspective projection PnP algorithm to calculate the first posture transformation matrix or the second posture transformation matrix includes: Calculate a rotation transformation matrix between the target RGB image and the related RGB image of the target RGB image; Calculate a translation transformation matrix between a related RGB image of the target RGB image and the target RGB image according to the depth image and the rotation transformation matrix; Based on the rotation transformation matrix and the translation transformation matrix, the first posture transformation matrix or the second posture transformation matrix is ​​generated between the related RGB image of the target RGB image and the target RGB image.

6. The method for calculating the position and posture according to claim 5, characterized in that: The step of calculating the translation transformation matrix between the related RGB image of the target RGB image and the target RGB image according to the depth image and the rotation transformation matrix includes: Eliminating mutually matching 2D-2D feature point pairs on the relevant RGB image of the target RGB image and the target RGB image according to the rotation transformation matrix to obtain eliminated 2D-2D feature point pairs; Converting the eliminated 2D-2D feature point pairs into 3D-2D feature point pairs according to the depth image; The translation transformation matrix between the target RGB image and the related RGB image of the target RGB image is calculated according to the 3D-2D feature point pairs.

7. The method for calculating the position and posture according to claim 3, characterized in that: The step of constructing an initial three-dimensional map of the current environment according to the first pose transformation matrix and the depth image includes: According to the first posture transformation matrix, projecting the 3D feature points on the depth image onto the next frame RGB image to generate projected 2D feature points; Calculating a reprojection error between the projected 2D feature point and the 2D feature point on the next frame RGB image; If the reprojection error is less than a preset error threshold, the 3D feature point on the depth image is used as a target map point; An initial three-dimensional map of the current environment is constructed according to the target map points.

8. The method for calculating the posture according to any one of claims 3 to 7, characterized in that: The updating of the initial three-dimensional map of the current environment according to the second posture transformation matrix and the depth image to construct a target three-dimensional map of the current environment includes: According to the second posture transformation matrix, the 3D feature points on the depth image are respectively projected onto an RGB image located before the target RGB image to generate projected 2D feature points; Calculating a reprojection error between the projected 2D feature point and a 2D feature point on an RGB image before the target RGB image; If the reprojection error is less than a preset error threshold, the 3D feature point on the depth image is used as a new target map point; The new target map point is added to the initial three-dimensional map to construct a target three-dimensional map of the current environment.

9. The method for calculating the posture according to claim 2, characterized in that: The calculating, according to the target three-dimensional map, the position and posture of the electronic device when collecting other RGB images after the next frame of RGB image within the preset initialization sliding window includes: The next frame of the next RGB image is used as the current frame, and the following target operations are performed: Generate a 3D-2D feature point pair matching the target three-dimensional map and the current frame according to the target three-dimensional map, the current frame and the RGB image before the current frame; Calculating the pose of the current frame based on the 3D-2D feature point pair; updating the target three-dimensional map according to the current frame, and using the updated three-dimensional map as a new target three-dimensional map; The next frame of the current frame is used as a new current frame, and the target operation is executed cyclically until the pose of the last frame of the RGB image in the preset initialization sliding window is calculated.

10. The method for calculating the position and posture according to claim 9, characterized in that: The generating, according to the target three-dimensional map, the current frame and the RGB image before the current frame, a 3D-2D feature point pair matching each other between the target three-dimensional map and the current frame comprises: Acquire mutually matching 2D feature point pairs from the current frame and an RGB image preceding the current frame; 3D feature points matching the 2D feature point pairs are obtained from map points in the target 3D map, and 3D-2D feature point pairs matching each other between the target 3D map and the current frame are generated based on the matched 3D feature points and the 2D feature point pairs.

11. The method for calculating the position and posture according to claim 9, characterized in that: If the current frame has a corresponding depth image, then updating the target three-dimensional map according to the current frame to generate an updated three-dimensional map includes: The target three-dimensional map is updated to generate an intermediate three-dimensional map according to the depth image corresponding to the current frame and the pose transformation matrix between the current frame and the RGB image before the current frame in a preset sliding window; The intermediate three-dimensional map is updated by using a triangulation method to generate the updated three-dimensional map.

12. The method for calculating the position and posture according to claim 9, characterized in that: If the current frame does not have a corresponding depth image, updating the target three-dimensional map according to the current frame to generate an updated three-dimensional map includes: The target three-dimensional map is updated by using a triangulation method to generate an updated three-dimensional map.

13. The method for calculating the position and posture according to claim 2, characterized in that: After calculating the position and posture of the electronic device when collecting other RGB images after the next frame of RGB image within the preset initialization sliding window according to the target three-dimensional map, the method further includes: Acquire the IMU data collected within the preset sliding window; Calculate the initialization information of the IMU according to the position and posture of each frame of the RGB image in the preset sliding window and the IMU data; the initialization information includes an initial velocity, a zero bias of the IMU, and a gravity vector of the IMU; The pose of the RGB image collected after the preset sliding window is calculated according to the initial pose, the target three-dimensional map and the initialization information of the IMU.

14. The method for calculating the position and posture according to claim 1, characterized in that: The method further comprises: If the depth image of the current environment is not collected within the preset initialization sliding window, the initial posture and the initial three-dimensional map are calculated according to the RGB image collected within the preset initialization sliding window; The posture of the electronic device when collecting the RGB image after the preset initialization sliding window is calculated according to the initial posture and the initial three-dimensional map.

15. The method for calculating the position and posture according to claim 1, characterized in that: The method further comprises: If the target RGB image corresponding to the depth image is the last frame image in the preset initialization sliding window, the initial pose and the initial three-dimensional map are calculated according to the RGB image collected in the preset initialization sliding window; The posture of the electronic device when collecting the RGB image after the preset sliding window is calculated according to the initial posture and the initial three-dimensional map.

16. A posture calculation device, characterized in that: The device comprises: An initial posture determination module is used to determine the posture of the electronic device when collecting the depth image as the initial posture if a depth image of the current environment is collected for the first time within a preset initialization sliding window; wherein the target RGB image corresponding to the depth image is not the last frame image within the preset initialization sliding window; and the acquisition frequency of collecting the depth image is lower than the acquisition frequency of collecting the target RGB image; The next frame RGB image posture determination module is used to determine the posture transformation matrix between the target RGB image and the next frame RGB image of the RGB image according to the target RGB image, the depth image and the next frame RGB image of the RGB image; and determine the posture of the electronic device when capturing the next frame RGB image according to the initial posture and the posture transformation matrix.

17. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor performs the steps of the posture calculation method according to any one of claims 1 to 15.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the posture calculation method according to any one of claims 1 to 15 are implemented.

Images