Position calculation method, device, storage medium and electronic device

Abstract

The present application discloses a posture calculation method, device, storage medium and electronic device, and relates to the field of equipment positioning technology. If it is determined that there is target feature information in the current frame color image, and it is confirmed that there is target historical feature information that matches the target feature information, it means that the historical frame color image has the same target feature information as the current frame color image, so the rotation component in the current posture of the electronic device can be calculated directly according to the target historical feature information and the target feature information in the current frame color image. Since the rotation component in the current posture is obtained based on the target historical feature information, that is, the error of the rotation component in the current posture is related to the error of the rotation component in the historical posture in the historical frame color image, the rotation component in the current posture does not depend on the correlation between the previous and next frame color images, so the cumulative error of calculating the rotation component in the current posture is reduced, and the accuracy of calculating the current posture is improved.

Description

Technical Field

[0001] The present application relates to the technical field of device positioning, and in particular to a posture calculation method, device, storage medium and electronic device. Background Art

[0002] With the rapid development of science and technology, people are increasingly using various electronic devices in their daily lives. Among them, indoor navigation and three-dimensional reconstruction of electronic devices have become one of the key research areas for those skilled in the art. The calculation of the current posture of electronic devices is a key part of indoor navigation and three-dimensional reconstruction, so it is necessary to provide a posture calculation solution with high accuracy. Summary of the invention

[0003] The present application provides a posture calculation method, device, storage medium and electronic device, which can solve the technical problem of low posture calculation accuracy in related technologies.

[0004] In a first aspect, an embodiment of the present application provides a posture calculation method, which is applied to an electronic device, and the method includes:

[0005] Acquire first plane information in a current frame depth image corresponding to a current frame color image; if it is determined based on the first plane information that there is target feature information in the current frame color image, confirm whether there is target historical feature information that matches the target feature information; if there is target historical feature information that matches the target feature information, calculate a rotation component in a current posture of the electronic device based on the target historical feature information and the target feature information; calculate a translation component in the current posture of the electronic device, and calculate the current posture of the electronic device based on the rotation component and the translation component.

[0006] In a second aspect, an embodiment of the present application provides a posture calculation device, which is applied to an electronic device, and the device includes:

[0007] A plane information acquisition module, used to acquire the first plane information in the current frame depth image corresponding to the current frame color image; a historical feature information determination module, used to confirm whether there is target historical feature information matching the target feature information if it is determined according to the first plane information that there is target feature information in the current frame color image; a rotation component calculation module, used to calculate the rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information if there is target historical feature information matching the target feature information; a current posture calculation module, used to calculate the translation component in the current posture of the electronic device, and calculate the current posture of the electronic device according to the rotation component and the translation component.

[0008] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the steps of the above-mentioned method.

[0009] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0010] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0011] The present application provides a posture calculation method. After obtaining the first plane information in the current frame depth image corresponding to the current frame color image, if it is determined according to the first plane information that there is target feature information in the current frame color image, and it is confirmed that there is target historical feature information matching the target feature information, it means that the historical frame color image has the same target feature information as the current frame color image. Since the target historical feature information is the feature information of the target feature information in the historical frame color image in a preset reference system, the rotation component in the current posture of the electronic device can be directly calculated according to the target historical feature information and the target feature information in the current frame color image, and the translation component in the current posture of the electronic device can also be calculated, and the current posture of the electronic device is calculated according to the rotation component and the translation component. Since the rotation component in the current posture is obtained based on the target historical feature information, that is, the error of the rotation component in the current posture is related to the error of the rotation component in the historical posture in the historical frame color image, and the rotation component in the current posture does not depend on the correlation between the previous and next frame color images, the cumulative error of calculating the rotation component in the current posture is reduced, thereby improving the accuracy of calculating the current posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 An exemplary system architecture diagram of a posture calculation method provided in an embodiment of the present application;

[0014] Figure 2 A flowchart of a posture calculation method provided in an embodiment of the present application;

[0015] Figure 3A schematic diagram of target feature information provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of historical target feature information provided by an embodiment of the present application;

[0017] Figure 5 A flowchart of a posture calculation method provided in another embodiment of the present application;

[0018] Figure 6 A flowchart of a posture calculation method provided in another embodiment of the present application;

[0019] Figure 7 A flowchart of a posture calculation method provided in another embodiment of the present application;

[0020] Figure 8 A schematic diagram of the structure of a posture calculation device provided in another embodiment of the present application;

[0021] Fig. 9 A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0023] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0024] Nowadays, industries such as mobile robots, unmanned driving, mobile indoor navigation and 3D reconstruction, and augmented reality are developing rapidly and have broad application prospects and commercial value in the future. These industries or applications are inseparable from a key technology - real-time positioning technology. The above mobile devices need centimeter-level self-positioning to know the position relationship between themselves and the environment in order to carry out subsequent tasks. Obviously, the existing positioning technology cannot meet the positioning accuracy requirements.

[0025] Based on the above needs, Simultaneous Localization and Mapping (SLAM) technology came into being. SLAM technology refers to the use of information provided by the mobile device's own sensors in a completely unknown environment, combined with corresponding algorithms to achieve centimeter-level positioning. At the same time, SLAM technology also includes environmental mapping while positioning. There are different ways to implement SLAM using different sensors. The current mainstream sensors are LiDAR SLAM, cameras (Visual SLAM) and inertial measurement units (Visual-inertial SLAM). On mobile phones or mobile robots, the most commonly installed sensor is the camera sensor. The positioning and mapping process of the camera body is completed through the image sequence collected by the camera, which is usually called visual SLAM technology.

[0026] At present, the algorithm framework of visual SLAM is basically mature. This technology has many advantages. It only uses cameras as external sensors to provide input for the algorithm, which greatly reduces the cost. The camera can provide rich scene information and is more conducive to high-precision positioning and mapping. However, there are still some problems in the application of visual SLAM in the industry. For example, the positioning in visual SLAM technology is to calculate the current posture of the electronic device or the posture of the camera in the electronic device. Generally, the visual positioning process uses the method of previous and next frames to achieve continuous positioning, that is, the posture corresponding to the current frame image is calculated based on the current frame image and the previous frame image. Since the posture calculated in the visual SLAM technology will have certain errors, the continuously accumulated posture calculation errors will cause serious positioning drift in the long-term positioning process.

[0027] Based on the above technical problems, an embodiment of the present application provides a posture calculation method, which uses feature information including planes to establish association constraints between the current frame and the historical frames, and then solves the posture of the electronic device, avoiding the cumulative errors caused by continuously calculating the posture between the previous and next frames in the related positioning algorithm.

[0028] See also Figure 1 , Figure 1 An exemplary system architecture diagram of a posture calculation method provided in an embodiment of the present application.

[0029] like Figure 1As shown, the system architecture may include an electronic device 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the electronic device 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links, for example, the wired communication link includes an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, or a microwave communication link.

[0030] The electronic device 101 can interact with the server 103 through the network 102 to receive messages from the server 103 or send messages to the server 103, or the electronic device 101 can interact with the server 103 through the network 102 to receive messages or data sent by other users to the server 103. The electronic device 101 can be hardware or software. When the electronic device 101 is hardware, it can be various electronic devices, including but not limited to smart watches, smart phones, tablet computers, laptop portable computers, desktop computers, and intelligent robots. When the electronic device 101 is software, it can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0031] The server 103 may be a business server that provides various services. It should be noted that the server 103 may be hardware or software. When the server 103 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or it may be implemented as a single server. When the server 103 is software, it may be implemented as multiple software or software modules (for example, for providing distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0032] It should be understood that Figure 1 The number of electronic devices, networks and servers in the figure is only illustrative, and any number of electronic devices, networks and servers may be used according to implementation requirements.

[0033] See also Figure 2 , Figure 2 A flowchart of a posture calculation method provided for an embodiment of the present application. The execution subject of the embodiment of the present application can be an electronic device that executes the posture calculation method, or a processor in an electronic device that executes the posture calculation method, or a posture calculation service in an electronic device that executes the posture calculation method. For the convenience of description, the specific execution process of the posture calculation method is introduced below by taking the execution subject being a processor in an electronic device as an example.

[0034] like Figure 2As shown, the pose calculation method includes:

[0035] S201. Obtain first plane information in a current frame depth image corresponding to a current frame color image.

[0036] When continuous positioning is achieved by using the method of previous and next frame color images, since the interval between the current frame color image and the previous frame color image is small, the pose obtained by calculating the constraint between the current frame color image and the previous frame color image will have a certain calculation error. When calculating the pose corresponding to the next frame color image, the constraint relationship between the next frame color image and the current frame color image and the pose corresponding to the current frame color image will be used. As a result, the error of the pose corresponding to the next frame color image will continue to accumulate on the basis of the error of the pose corresponding to the current frame color image. Therefore, the continuously accumulated pose calculation error will cause serious positioning drift in the long-term positioning process.

[0037] Then a feasible solution is that if after the electronic device moves its position, if it is determined that the scene corresponding to the current frame color image has the same features as those in the scene corresponding to a certain historical frame color image, then it can be considered that the electronic device has moved back to the vicinity of the previous position (that is, the position of the electronic device corresponding to a certain historical frame color image), that is, the current frame color image is associated with a certain historical frame color image, and the electronic device calculates the corresponding posture each time it moves to a position by default, that is, the historical postures corresponding to the electronic device at the previous position have been calculated, then the electronic device can be set so that after the electronic device extracts the preset features in the historical frame color image, the features can be transformed into a unified preset coordinate system based on the rotation component of the historical posture corresponding to the historical frame color image. The preset coordinate system can be a coordinate system such as the world coordinate system, so that after the electronic device extracts the same preset features in the subsequent frame color image, it can check in the unified preset coordinate system whether it is the same preset feature. If it is the same preset feature, the features transformed to the unified preset coordinate system and the preset features can be directly used to calculate the rotation component of the current posture corresponding to the current frame color image.

[0038] Since the current posture corresponding to the current frame color image is calculated using the historical posture corresponding to the historical frame color image with the same features, where the historical frame color image is a frame color image before the current frame color image, the error of the posture corresponding to the current frame color image is only related to the error of the posture corresponding to the historical frame color image. Therefore, the error of the posture corresponding to any frame color image between the historical frame color image and the current frame color image will not affect the calculation of the posture corresponding to the current frame color image, that is, the error of the current posture corresponding to the current frame color image calculated using the historical posture corresponding to the historical frame color image with the same features is much smaller than the error of the posture calculated using the current frame color image and the previous and next frame color images.

[0039] Based on the above ideas, it is necessary to determine whether there is a preset feature in the current frame color image that is the same as that in a historical frame color image, that is, the historical frame color image cannot be any frame color image before the current frame color image. Since the plane in the scene is easier to distinguish and identify than a visual point in the scene, a feasible implementation method for determining the historical frame color image corresponding to the current frame color image is to determine whether there is a preset feature (that is, the target feature information in this application) based on the relationship between the planes in the current scene corresponding to the current frame color image of the electronic device, and then judge the historical frame color image corresponding to the previous frame color image.

[0040] Therefore, it is first necessary to obtain the first plane information in the current scene corresponding to the current frame color image of the electronic device. In an embodiment of the present application, an RBG camera may be provided in the electronic device to capture a color image of the scene in which the electronic device is located through the RBG camera. In addition, an RGBD camera may be provided in the electronic device to capture a depth image of the scene in which the electronic device is located through the RGBD camera. The depth image can provide depth information of the scene in which the electronic device is located to quickly restore the three-dimensional structure of the current scene.

[0041] Since the RGBD camera is set in the electronic device and the lens of the RGBD camera has a certain shooting range and shooting angle, after the RGBD camera shoots a frame of depth image, the depth image will correspond to the current scene of the electronic device, that is, all planes in the current frame depth image corresponding to the scene of the electronic device or planes that meet preset plane conditions can be extracted based on the current frame depth image shot by the RGBD camera in the electronic device. These planes can be considered as the first plane information in the current frame depth image or a part of the first plane information, that is, the first plane information at least includes the above-mentioned plane extracted from the current frame depth image, wherein plane extraction on the current frame depth map is faster and the obtained plane accuracy is also higher.

[0042] Optionally, in an embodiment of the present application, the method for extracting all planes in the current frame depth image may not be limited. For example, a feasible implementation is to back-project all pixel points on the current frame depth map into three-dimensional space according to the intrinsic parameter matrix of the depth camera and the corresponding depth value to form a three-dimensional point cloud of the current frame depth image. A plane detection algorithm is used in the three-dimensional point cloud to extract the plane (the plane detection algorithm here is not specifically limited, and can be a method of fitting a plane model in the point cloud using the Ransac algorithm, or a method of performing region growing after calculating the normal vector information of the three-dimensional point).

[0043] The mathematical representation of the plane in the extracted depth image of the current frame is {n, dis}, where n is a three-dimensional vector representing the normal vector of the plane; and dis represents the distance from the origin of the coordinate system corresponding to the three-dimensional space to the plane.

[0044] In the embodiment of the present application, the color image and the depth image are aligned in time and space, so the first plane information in the depth image of the current frame can be considered as the first plane information in the color image of the current frame.

[0045] S202: If it is determined according to the first plane information that there is target feature information in the current frame color image, then confirm whether there is target historical feature information that matches the target feature information.

[0046] In plane relationships, the most obvious relationships are the angles between planes and the quantitative relationships between planes. Therefore, after obtaining the first plane information in the current frame color image of the electronic device, the angle relationship between the planes can be determined based on the normal vectors of the respective planes in the first plane information. Then, based on the angle relationship between the planes and the quantitative relationship between the planes, it is determined whether there is target feature information in the corresponding scene of the electronic device. In an embodiment of the present application, the color image and the depth image are aligned in time and space, so the target feature information is the basis for determining whether the current frame color image is associated with a certain historical frame color image. In order to ensure the accuracy of determining whether the current frame color image is associated with the historical frame color image, it is necessary to ensure that the target feature information is as recognizable and distinguishable as possible.

[0047] In practical applications, the target feature information may include a plane that satisfies a preset angle relationship and a preset quantity relationship, wherein the preset angle relationship and the preset quantity relationship may be set according to a specific method for calculating the posture of the electronic device.

[0048] Since the rotation component in the posture of the electronic device is a variable in the three-dimensional coordinate system, and in the subsequent calculation of the rotation component in the posture of the electronic device, it is first necessary to transform the target feature information into the preset reference system to match the historical feature information corresponding to the target feature information in the preset reference system, and then it is necessary to directly use the historical feature information and the target feature information to calculate the rotation component in the posture of the electronic device, and the preset reference system needs to be one of the three-dimensional coordinate systems. Therefore, in an embodiment of the present application, the target feature information may include three planes, so that the target feature information can be transformed into the three-dimensional coordinate system, and the target feature information can be uniquely determined in the three-dimensional coordinate system.

[0049] See also Figure 3 , Figure 3 A schematic diagram of target feature information provided in an embodiment of the present application.

[0050] If the target feature information includes three planes that satisfy the preset angle relationship, such as Figure 3 As shown, after three planes (whose normal vectors are n1, n2 and n3) are detected in the current frame depth image, the angles between the three planes can be calculated respectively to determine whether the three planes meet the preset angle relationship. Specifically, the angles between the normal vectors corresponding to the three planes can be calculated to calculate the angles between the three planes respectively. The formula for calculating the angles between the normal vectors corresponding to the three planes is:

[0051] θ1 = arccos(n1, n2);

[0052] θ2 = arccos(n2, n3);

[0053] θ3 = arccos(n1,n3);

[0054] Among them, θ1, θ2 and θ3 respectively represent the angles between the three planes. If the three angles satisfy the preset angle relationship, it can be determined that the three planes constitute the target feature information.

[0055] Furthermore, if the target feature information includes three planes that satisfy a preset angle relationship, since features composed of planes such as wall corners in real scenes are relatively special, the target feature information can be set to include three mutually perpendicular planes. Then θ1, θ2 and θ3 respectively represent the angles between the three planes. If θ1, θ2 and θ3 are all 90 degrees, or θ1, θ2 and θ3 are all within a preset range near 90 degrees (for example, between 88 and 92 degrees), it can be determined that these three planes constitute the target feature information.

[0056] If it is determined according to each plane that there is target feature information in the current frame depth image, the target feature information can be represented by three normal quantities of the target feature information. In the embodiment of the present application, the letter SF (Structure Feature) is used to represent the feature information, so the above target feature information can be represented as follows:

[0057] SF:[n1,n2,n3].

[0058] Since the target feature information corresponds to special positions such as corners in actual scenes, which are not very common, it can be determined based on the target feature information whether the current frame color image is associated with a certain historical frame color image.

[0059] Specifically, in the embodiment of the present application, a historical feature information database is pre-established, and at least one historical feature information is stored in the database. Each historical feature information is obtained by transforming the target feature information in the historical frame color image into a preset reference system according to the rotation component of the historical posture corresponding to the historical frame color image. That is, in the embodiment of the present application, the posture corresponding to each frame of color image is calculated, so each historical frame color image also corresponds to a historical posture. Then, when it is determined that there is target feature information in the historical frame color image, and it is determined that the target feature information appears for the first time, the target feature information in the historical frame color image will be transformed into a preset reference system according to the rotation component of the historical posture corresponding to the target feature information in the historical frame color image to obtain the historical feature information corresponding to the historical frame color image.

[0060] Among them, the purpose of transforming the target feature information in the historical frame color image into the preset reference system is that each time the electronic device obtains the color image, the position of the electronic device may be different. Therefore, although the target feature information can be determined in two different frames of color images, it is very likely that they are not the target feature information at the same position. Therefore, it is necessary to uniformly transform the target feature information into the preset reference system to obtain the historical feature information. Then, the position of each historical feature information in the preset reference system is uniquely determined, so as to judge whether the target feature information in different frames of color images is the target feature information at the same position.

[0061] After determining that there is target feature information in the current frame depth image of the electronic device, it is also necessary to search the historical feature information library to see whether there is matching target historical feature information under the preset reference system to confirm whether there is target historical feature information that matches the target feature information.

[0062] Optionally, a method for finding target historical feature information matching the target feature information from the historical feature information library is to first transform the target feature information to the preset reference system according to the rotation component of the current frame color image to the preset reference system, wherein the rotation component of the current frame color image to the preset reference system can be a temporary rotation component, and the rotation component can be a rough estimate, for example, it can be roughly obtained by some hardware equipment or algorithm, and the rotation component of the posture in the previous frame color image can also be directly used as the temporary rotation component; then obtain the plane angle relationship between the target feature information after being transformed to the preset reference system and the historical feature information, that is, calculate the plane angle between the plane included in the target feature information after being transformed to the preset reference system and the plane included in each historical feature information in the historical feature information library; finally, if the plane angle relationship satisfies the preset plane angle relationship, it is confirmed that the target historical feature information matching the target feature information is found from the historical feature information library, that is, it is confirmed that there is target historical feature information matching the target feature information, and the historical feature information corresponding to the preset plane angle relationship is determined as the target historical feature information, wherein the preset plane angle relationship can be set according to actual needs.

[0063] For example, the preset plane angle relationship can be set to a plane angle less than 10 degrees. If the target feature information includes three planes that satisfy the preset angle relationship, if the plane included in the calculated target feature information after being transformed into the preset reference system is less than 10 degrees from the plane angle included in a certain historical feature information in the historical feature information library, then it can be considered that target historical feature information matching the target feature information is found from the historical feature information library, and it can be confirmed that there is target historical feature information matching the target feature information.

[0064] S203: If there is target historical feature information that matches the target feature information, calculate the rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information.

[0065] If it is determined based on the first plane information that there is target feature information in the current frame color image, and target historical feature information that matches the target feature information in a preset reference system is found from the historical feature information library, that is, it is determined that there is target historical feature information that matches the target feature information, then it means that the current frame color image corresponding to the target feature information is associated with the historical frame color image corresponding to the target historical feature information, that is, the determined target feature information in the current frame color image and the determined target feature information in the historical frame color image are feature information at the same position.

[0066] In the embodiment of the present application, the target feature information composed of the plane is more distinguishable and recognizable than the visual feature points, and can quickly and stably identify and match the same feature information observed between different frames of color images.

[0067] Since the target historical feature information is obtained by transforming the target feature information into a preset reference system according to the rotation component of the historical posture corresponding to the target feature information, the rotation component in the current posture of the electronic device can be directly calculated based on the target historical feature information and the target feature information. Specifically, the rotation component in the current posture of the electronic device can be calculated based on the transformation relationship between the target historical feature information and the target feature information in the current frame color image.

[0068] See also Figure 4 , Figure 4 A schematic diagram of historical target feature information provided in an embodiment of the present application.

[0069] like Figure 4 As shown, there are three historical feature information in the historical feature information library, namely the first historical feature information (frame1), the sixth historical feature information (frame6) and the tenth historical feature information (frame10), wherein the first historical feature information is obtained by transforming the target feature information in the first frame color image into a preset reference system according to the rotation component of the historical posture corresponding to the first frame color image; the sixth historical feature information is obtained by transforming the target feature information in the sixth frame color image into a preset reference system according to the rotation component of the historical posture corresponding to the first frame color image; the tenth historical feature information is obtained by transforming the target feature information in the tenth frame color image into a preset reference system according to the rotation component of the historical posture corresponding to the first frame color image.

[0070] Then, if it is determined according to each plane that there is target feature information in the current frame color image, the target historical feature information that matches the target feature information in the preset reference system is found in the historical feature information database as the sixth historical feature information, then the relationship between the target historical feature information and the target feature information can be obtained:

[0071] SF frame6 =R*SF cur ;

[0072] Among them, SF frame6 is the sixth historical feature information, R is the rotation component of the current posture corresponding to the current frame color image, SF cur The target feature information corresponding to the current frame color image.

[0073] Therefore, based on the relationship between the above target historical feature information and the target feature information, the calculation formula for the rotation component in the current posture of the electronic device can be obtained as follows:

[0074] R=SF frame6 *SF cur -1 ;

[0075] Then, based on the transformation relationship between the target historical feature information and the target feature information in the current frame color image, the rotation component in the current posture of the electronic device can be calculated, that is, the product of the target historical feature information and the inverse transformation of the target feature information in the current frame color image is used as the rotation component in the current posture of the electronic device.

[0076] S204, calculating the translation component in the current posture of the electronic device, and calculating the current posture of the electronic device according to the rotation component and the translation component.

[0077] Since the current posture of the electronic device includes at least a rotation component and a translation component, after calculating the rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information, the translation component in the current posture of the electronic device can be calculated, wherein the method for calculating the translation component in the current posture of the electronic device may not be limited. Furthermore, after obtaining the rotation component and the translation component of the electronic device, the rotation component and the translation component of the electronic device may be processed in a related manner, for example, the rotation component and the translation component of the electronic device may be transformed, etc., so as to calculate the current posture of the electronic device according to the rotation component and the translation component.

[0078] In an embodiment of the present application, after obtaining the first plane information in the current frame depth image corresponding to the current frame color image, if it is determined according to the first plane information that there is target feature information in the current frame color image, and it is confirmed that there is target historical feature information that matches the target feature information, it means that the historical frame color image has the same target feature information as the current frame color image. Since the target historical feature information is the feature information of the target feature information in the historical frame color image in a preset reference system, the rotation component in the current posture of the electronic device can be directly calculated based on the target historical feature information and the target feature information in the current frame color image, and then the translation component in the current posture of the electronic device can also be calculated, and the current posture of the electronic device can be calculated based on the rotation component and the translation component. Since the rotation component in the current posture is obtained based on the target historical feature information, that is, the error of the rotation component in the current posture is related to the error of the rotation component in the historical posture in the historical frame color image, and the rotation component in the current posture does not depend on the correlation between the previous and next frame color images, the cumulative error of calculating the rotation component in the current posture is reduced, thereby improving the accuracy of calculating the current posture.

[0079] See also Figure 5 , Figure 5 A flowchart of a posture calculation method provided in another embodiment of the present application.

[0080] like Figure 5 As shown, the method includes:

[0081] S501. Obtain first plane information in a current frame depth image corresponding to a current frame color image.

[0082] Regarding step S501, please refer to the description in step S201, which will not be repeated here.

[0083] S502: If it is determined according to the first plane information that there is target feature information in the current frame color image, then confirm whether there is target historical feature information that matches the target feature information.

[0084] Regarding step S502, please refer to the description in step S202, which will not be repeated here.

[0085] S503: If there is target historical feature information that matches the target feature information, calculate the rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information.

[0086] Regarding step S503, please refer to the description in step S203, which will not be repeated here.

[0087] S504: Solve the translation component in the current posture of the electronic device based on the rotation component in the current posture, and calculate the current posture of the electronic device according to the rotation component and the translation component.

[0088] Since the target historical feature information is obtained by transforming the target feature information in the historical frame color image into a preset reference system based on the rotation component of the historical posture corresponding to the historical frame color image, only the rotation component in the current posture of the electronic device can be calculated based on the target historical feature information and the target feature information. The target historical feature information does not involve the translation component in the historical posture, so the translation component in the current posture cannot be directly obtained through the target historical feature information and the target feature information.

[0089] After obtaining the rotation component in the current posture of the electronic device, a constraint relationship can be established based on the current frame color image and the reference frame color image, and then the rotation component in the current posture of the electronic device is used as a known quantity to obtain the translation component in the current posture.

[0090] Specifically, in one embodiment, based on the constraint relationship established between the current frame color image and the reference frame color image, the current frame color image of the electronic device and the reference frame color image corresponding to the current frame color image can be obtained first, wherein the reference frame color image corresponding to the current frame color image can be a color image near the current frame color image, for example, the reference frame color image can be a color image before the current frame color image, or can be a color image several frames before the current frame color image, and the number of reference frame color images can be unlimited; then, a reprojection error of feature points between the current frame color image and the reference frame color image is constructed.

[0091] Specifically, in the process of constructing the reprojection error of the feature points between the current frame color image and the reference frame color image, it is first necessary to extract the feature points corresponding to the current frame color image and the reference frame color image. The type of feature points here is not limited and can be any commonly used visual feature points, such as fast corner points, Harris corner points, ORB feature points, etc.; then determine the corresponding matching feature points in the current frame color image and the reference frame color image, and finally construct the reprojection error of the feature points between the current frame color image and the reference frame color image based on the matching feature points.

[0092] For example, it is known that there is a feature point a(u1, v1) in the reference frame color image i, and there is a feature point b(u2, v2) in the current frame color image j, where feature point a(u1, v1) and feature point b(u2, v2) are a pair of matching feature points, and the depth corresponding to feature point a is d a , the depth corresponding to feature point b is d b , then the classic reprojection error between feature point a and feature point b is as follows:

[0093] error 重投影 =T ij d a k -1 (u1,v1,1)-k -1 (u2,v2,1) (1);

[0094] In formula (1), T ij is the pose between the reference frame color image i and the current frame color image j (including the rotation component and the translation component), k -1 is the intrinsic parameter matrix of the RBG camera.

[0095] After obtaining the reprojection error, optimize T ij , k -1 Two state variables, error 重投影 Minimize to obtain the optimal posture and optimal feature point depth value. Then, in the process of optimizing the reprojection error, it is necessary to optimize Tij , k -1 Two state variables, error 重投影 Minimize to obtain the optimal posture and optimal feature point depth value.

[0096] In the embodiment of the present application, since the rotation component in the current posture has been obtained, the rotation component in the current posture can be used as a known quantity to optimize T ij The translation component in k -1 Two state variables, error 重投影 Minimum, in order to obtain the optimal translation component in the posture and the optimal feature point depth value, that is, based on the rotation component and the reprojection error in the current posture, the translation component in the current posture of the electronic device is solved, and then the current posture of the electronic device can be calculated according to the rotation component and the translation component.

[0097] When local multi-frame color images are jointly optimized, there may be hundreds of feature points on each frame, and thousands of the above-mentioned reprojection errors will be constructed. This means that the depth values ​​of many feature points need to be optimized during the joint optimization, which will make the scale of the optimization problem very large and the optimization and solution of the optimal posture will take a long time.

[0098] In an embodiment of the present application, in addition to extracting image feature point information, plane information in the image can also be extracted. In order to solve the problems in the above-mentioned optimization process, a feasible way is to use plane information to represent depth in the process of constructing the reprojection error of the feature points between the current frame color image and the reference frame color image, thereby reducing the dimension of the state variables to be optimized and reducing the time consumption of the optimization solution process.

[0099] See also Figure 6 , Figure 6 A flowchart of a posture calculation method provided in another embodiment of the present application.

[0100] like Figure 6 As shown, the method for constructing the reprojection error of the feature points between the current frame color image and the reference frame color image may include:

[0101] S601 : Determine target feature points among feature points of a reference frame color image according to feature points of a reference frame color image and second plane information of a reference depth image corresponding to the reference frame color image.

[0102] Specifically, in the process of constructing the reprojection error of the feature points between the current frame color image and the reference frame color image, the feature points of the reference frame color image corresponding to the current frame color image can be first obtained, and then the target feature points among the feature points of the reference frame color image are determined based on the feature points of the reference frame color image and the second plane information in the reference depth image corresponding to the reference frame color image, wherein the target feature points are within a preset range of the same plane to be optimized of the reference depth image.

[0103] S602: construct a reprojection error of feature points between the current frame color image and the reference frame color image based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device.

[0104] In order to reduce the dimension of the state variables to be optimized, in an embodiment of the present application, plane information is used to represent depth, that is, based on the plane parameters to be optimized and the posture to be optimized of the electronic device, a reprojection error of the feature points between the current frame color image and the reference frame color image is constructed.

[0105] For example, it is known that there is a feature point a(u1, v1) in the reference frame color image i that falls within a preset range of the plane (it can be considered that the feature point falls on the plane), and the plane is p1{n1, dis1}, then the feature point a(u1, v1) can be expressed by the following plane equation:

[0106]

[0107] Furthermore, d can be expressed as a This variable to be optimized is represented by the plane parameters n1 and dis1:

[0108]

[0109] In formula (3), it means that any feature point that falls within the preset range of the plane p1{n1, dis1} can be expressed by the parameters n1 and dis1 of the plane p1{n1, dis1}.

[0110] Then if there are m target feature points in the reference frame color image that fall within the preset range of the plane to be optimized p1{n1, dis1}, then if the reprojection error constraint of formula (1) is adopted, in addition to optimizing the posture, the depth values ​​of these m target feature points need to be optimized. In the embodiment of the present application, the depth values ​​of these m target feature points are expressed by the parameters n1 and dis1 of the plane to be optimized p1{n1, dis1}, and are substituted into the reprojection error of formula (1), that is, formula (3) is substituted into the reprojection error of formula (1), and the joint reprojection error of the m target feature points on the plane to be optimized p1{n1, dis1} can be obtained after simplifying the plane information:

[0111]

[0112] Then based on the rotation component and reprojection error in the current posture, solving the translation component in the current posture of the electronic device can specifically include: obtaining the target matching feature points corresponding to the target feature points in the current frame color image, solving the minimum value of the reprojection error based on the target matching feature points and the rotation component, and obtaining the translation component in the current posture of the electronic device. That is, after obtaining the reprojection error, the target matching feature points corresponding to the target feature points in the current frame color image can be extracted according to the target feature points, and then the minimum value of the reprojection error can be solved based on the target matching feature points and the rotation component, that is, the rotation component is used as a known quantity, and the minimum value of the reprojection error is solved based on the target matching feature points, so that the translation component in the current posture of the electronic device can be obtained.

[0113] It can be seen from formula (4) that compared with the previous projection error optimization process, which required optimizing the posture and optimizing the depth values ​​of multiple feature points, it has become necessary to optimize only the posture and the parameters n1 and dis1 of p1{n1, dis1}. This design greatly reduces the dimension of the optimization variables, makes the optimization problem lighter, greatly reduces the scale of the optimization problem, improves the optimization efficiency, and greatly improves the optimization speed.

[0114] See also Figure 7 , Figure 7 A flowchart of a posture calculation method provided in another embodiment of the present application.

[0115] like Figure 7 As shown, the method includes:

[0116] S701. Obtain first plane information in a current frame depth image corresponding to a current frame color image.

[0117] S702: If it is determined according to the first plane information that there is target feature information in the current frame color image, then confirm whether there is target historical feature information that matches the target feature information.

[0118] Regarding steps S701 to S702, please refer to the description in steps S201 to S202, which will not be repeated here.

[0119] S703. If there is no target historical feature information that matches the target feature information, obtain the current frame color image of the electronic device and the reference frame color image corresponding to the current frame color image, and construct a reprojection error of feature points between the current frame color image and the reference frame color image.

[0120] If it is determined based on each plane that there is target feature information in the current frame color image, and no target historical feature information that matches the target feature information in the preset reference system is found in the historical feature information library, that is, it is determined that there is no target historical feature information that matches the target feature information, then it means that the target feature information in the current frame color image is the target feature information that appears for the first time. At this time, the current frame color image of the electronic device and the reference frame color image corresponding to the current frame color image can be directly obtained to construct the reprojection error of the feature points between the current frame color image and the reference frame color image.

[0121] Among them, constructing a reprojection error of feature points between the current frame color image and the reference frame color image includes: determining target feature points among the feature points of the reference frame color image according to the feature points of the reference frame color image and the second plane information in the reference depth image corresponding to the reference frame color image, wherein the target feature points are within a preset range of the same plane to be optimized of the reference depth image; constructing a reprojection error of feature points between the current frame color image and the reference frame color image based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device.

[0122] For a detailed description of constructing the reprojection error of the feature points between the current frame color image and the reference frame color image, please refer to the records in the above steps S601 to S602, which will not be repeated here.

[0123] S704 , solving the rotation component and the translation component in the current posture of the electronic device based on the reprojection error, and calculating the current posture of the electronic device according to the rotation component and the translation component.

[0124] Since the target feature information in the current frame color image appears for the first time, the rotation component in the current posture cannot be obtained based on the target feature information at this time. It is necessary to directly solve the rotation component and translation component in the current posture of the electronic device based on the reprojection error.

[0125] Specifically, the rotation component and the translation component in the current posture of the electronic device are solved based on the reprojection error, including: obtaining the target matching feature points corresponding to the target feature points in the current frame color image, solving the minimum value of the reprojection error based on the target matching feature points, and obtaining the rotation component and the translation component in the current posture of the electronic device, and then calculating the current posture of the electronic device based on the rotation component and the translation component.

[0126] Optionally, after solving the rotation component and the translation component in the current posture of the electronic device based on the reprojection error, historical feature information matching the target feature information can be obtained according to the target feature information of the current frame color image, and the historical feature information can be saved.

[0127] Specifically, the target feature information of the current frame color image can be transformed into a preset reference system according to the rotation component in the current posture of the electronic device, and the historical feature information of the current frame color image can be obtained, and the historical feature information of the current frame color image can be saved in the historical feature information library, so as to facilitate the calculation of the rotation component in the posture of subsequent frame images. Since the historical feature information library is constructed and continuously updated, the feature information is used to establish the structural association constraint between the current frame and the historical frame, and then the drift-free rotation is solved, which avoids the cumulative error caused by the continuous calculation of the posture between the previous and next frames in the positioning algorithm, and can achieve more accurate and robust positioning without increasing the burden of the positioning system optimization problem.

[0128] Optionally, in traditional positioning algorithms, point features are generally used to calculate pose. In the embodiments of the present application, surface features with higher dimensions than point features are used to calculate pose. Furthermore, higher-dimensional features, such as semantic features, can be introduced to increase the association constraints between different image frames, which can provide more precise feature matching, thereby obtaining a more accurate pose and further improving the positioning accuracy of the world SLAM algorithm.

[0129] Optionally, plane and scene structure information can also be used to make the construction of the environment map more accurate. Specifically, the use of corresponding plane and scene structure information in the embodiments of the present application is all for the positioning link in the SLAM process. Since plane and scene structure information are also very helpful for the environment mapping environment in the SLAM process, for example, if a certain area in the known scene is a plane, then a plane model can be used to perform three-dimensional reconstruction of the area, and it is no longer necessary to use a three-dimensional space point to construct a small grid, which can make the reconstructed map smoother. At the same time, if there is feature information, the constraints of the feature information can be added to the map, so that the reconstructed map will be strictly vertical in the area where the feature information exists, and there will be no angle error.

[0130] In an embodiment of the present application, when the rotation component in the current posture cannot be obtained based on the target feature information, the reprojection error of the feature points between the current frame color image and the reference frame color image can be constructed based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device. This can make the optimization problem more lightweight in the process of obtaining the rotation component and the translation component in the current posture, greatly reduce the scale of the optimization problem, improve the optimization efficiency, and greatly improve the optimization speed.

[0131] Optionally, in another embodiment of the present application, as long as the plane information extracted from the current frame depth image corresponding to the current frame color image of the electronic device includes at least one plane, even if the existence of target feature information in the current frame color image cannot be determined based on the plane information, the target feature points in the feature points of the reference frame color image can still be determined based on the feature points of the reference frame color image and the plane information in the reference depth image corresponding to the reference frame color image, and the target feature points are within a preset range of the same plane to be optimized in the reference depth image; then, based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device, a reprojection of the feature points between the current frame color image and the reference frame color image is constructed. Error, wherein the specific description of the reprojection error of the feature points between the current frame color image and the reference frame color image can be referred to the records in the above steps S601 to S602, which will not be repeated here; finally, the target matching feature points corresponding to the target feature points in the current frame color image are obtained, and the minimum value of the reprojection error is solved based on the target matching feature points to obtain the rotation component and the translation component in the current posture of the electronic device, and then the current posture of the electronic device is calculated according to the rotation component and the translation component. Similarly, the optimization problem in the process of obtaining the rotation component and the translation component in the current posture can be made lighter, which greatly reduces the scale of the optimization problem, improves the optimization efficiency, and greatly improves the optimization speed.

[0132] See also Figure 8 , Figure 8 A schematic structural diagram of a posture calculation device provided in accordance with another embodiment of the present application.

[0133] like Figure 8 As shown, the posture calculation device 800 is applied to an electronic device, including:

[0134] The plane extraction module 810 is used to obtain first plane information in the current frame depth image corresponding to the current frame color image.

[0135] The historical feature information determination module 820 is used to confirm whether there is target historical feature information matching the target feature information if it is determined according to the first plane information that there is target feature information in the current frame color image.

[0136] The rotation component calculation module 830 is used to calculate the rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information if there is target historical feature information that matches the target feature information.

[0137] The current posture calculation module 840 is used to calculate the translation component in the current posture of the electronic device, and calculate the current posture of the electronic device according to the rotation component and the translation component.

[0138] Optionally, the target historical feature information is obtained by transforming the target feature information in the historical frame color image into a preset reference system according to the rotation component of the historical posture corresponding to the historical frame color image;

[0139] The rotation component calculation module 830 is also used to calculate the rotation component in the current posture of the electronic device according to the transformation relationship between the target historical feature information and the target feature information in the current frame color image.

[0140] Optionally, the historical feature information determination module 820 is also used to obtain the plane angle relationship between the target feature information and the historical feature information after the target feature information is transformed into a preset reference system; if the plane angle relationship satisfies the preset plane angle relationship, it is confirmed that there is target historical feature information that matches the target feature information, and the historical feature information corresponding to the preset plane angle relationship is determined as the target historical feature information.

[0141] Optionally, the current posture calculation module 840 is further used to solve the translation component in the current posture of the electronic device based on the rotation component in the current posture.

[0142] Optionally, the current posture calculation module 840 is also used to obtain a current frame color image of the electronic device and a reference frame color image corresponding to the current frame color image, construct a reprojection error of feature points between the current frame color image and the reference frame color image; based on the rotation component and the reprojection error in the current posture, solve the translation component in the current posture of the electronic device.

[0143] Optionally, the current posture calculation module 840 is also used to determine the target feature points among the feature points of the reference frame color image based on the feature points of the reference frame color image and the second plane information in the reference depth image corresponding to the reference frame color image, and the target feature points are within a preset range of the same plane to be optimized of the reference depth image; based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device, construct the reprojection error of the feature points between the current frame color image and the reference frame color image.

[0144] Optionally, the current posture calculation module 840 is also used to obtain the target matching feature points corresponding to the target feature points in the current frame color image, solve the minimum value of the reprojection error based on the target matching feature points and the rotation component, and obtain the translation component in the current posture of the electronic device.

[0145] Optionally, the posture calculation device 800 further includes:

[0146] The first pose calculation module is used to obtain the current frame color image and the reference frame color image corresponding to the current frame color image if there is no target historical feature information matching the target feature information, and construct a reprojection error of the feature points between the current frame color image and the reference frame color image.

[0147] The second posture calculation module is used to solve the rotation component and the translation component in the current posture of the electronic device based on the reprojection error.

[0148] Optionally, the first posture calculation module is also used to determine the target feature points among the feature points of the reference frame color image based on the feature points of the reference frame color image and the second plane information in the reference depth image corresponding to the reference frame color image, and the target feature points are within a preset range of the same plane to be optimized of the reference depth image; based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device, construct a reprojection error of the feature points between the current frame color image and the reference frame color image.

[0149] Optionally, the second posture calculation module is also used to obtain the target matching feature points corresponding to the target feature points in the current frame color image, solve the minimum value of the reprojection error based on the target matching feature points, and obtain the rotation component and translation component in the current posture of the electronic device.

[0150] Optionally, the posture calculation device 800 further includes:

[0151] The historical feature information database updating module is used to obtain historical feature information matching the target feature information according to the target feature information of the current frame color image, and to save the historical feature information.

[0152] In an embodiment of the present application, a posture calculation device includes: a plane information acquisition module, which is used to acquire first plane information in a current frame depth image corresponding to a current frame color image; a historical feature information determination module, which is used to confirm whether there is target historical feature information matching the target feature information if it is determined that there is target feature information in the current frame color image according to the first plane information; a rotation component calculation module, which is used to calculate the rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information if there is target historical feature information matching the target feature information; and a current posture calculation module, which is used to calculate the translation component in the current posture of the electronic device, and calculate the current posture of the electronic device according to the rotation component and the translation component. After obtaining the first plane information in the current frame depth image corresponding to the current frame color image, if it is determined according to the first plane information that there is target feature information in the current frame color image, and it is confirmed that there is target historical feature information that matches the target feature information, it means that the historical frame color image has the same target feature information as the current frame color image. Since the target historical feature information is the feature information of the target feature information in the historical frame color image in a preset reference system, the rotation component in the current posture of the electronic device can be directly calculated based on the target historical feature information and the target feature information in the current frame color image, and then the translation component in the current posture of the electronic device can also be calculated, and the current posture of the electronic device can be calculated based on the rotation component and the translation component. Since the rotation component in the current posture is obtained based on the target historical feature information, that is, the error of the rotation component in the current posture is related to the error of the rotation component in the historical posture in the historical frame color image, and the rotation component in the current posture does not depend on the correlation between the previous and next frame color images, the cumulative error of calculating the rotation component in the current posture is reduced, thereby improving the accuracy of calculating the current posture.

[0153] An embodiment of the present application further provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of any method in the above embodiments.

[0154] For further information, see Fig. 9 , Fig. 9 A schematic diagram of the structure of an electronic device is provided for an embodiment of the present application. Fig. 9 As shown, the electronic device 900 may include: at least one central processor 901 , at least one network interface 904 , a user interface 903 , a memory 905 , and at least one communication bus 902 .

[0155] The communication bus 902 is used to realize the connection and communication between these components.

[0156] The user interface 903 may include a display screen (Display) and a camera (Camera), and the optional user interface 903 may also include a standard wired interface and a wireless interface.

[0157] The network interface 904 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0158] Among them, the central processing unit 901 may include one or more processing cores. The central processing unit 901 uses various interfaces and lines to connect various parts within the entire electronic device 900, and executes various functions and processes data of the electronic device 900 by running or executing instructions, programs, code sets or instruction sets stored in the memory 905, and calling data stored in the memory 905. Optionally, the central processing unit 901 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The central processing unit 901 can integrate one or more combinations of central processing units (Central Processing Unit, CPU), graphics processing units (Graphics Processing Unit, GPU) and modems. Among them, the CPU mainly processes operating systems, user interfaces, and applications; the GPU is responsible for rendering and drawing the content that needs to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the central processing unit 901, and can be implemented separately through a chip.

[0159] Among them, the memory 905 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 905 includes a non-transitory computer-readable storage medium. The memory 905 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 905 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 905 may also be optionally at least one storage device located away from the aforementioned central processor 901. As Fig. 9 As shown, the memory 905 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a posture calculation program.

[0160] exist Fig. 9 In the electronic device 900 shown, the user interface 903 is mainly used to provide an input interface for the user and obtain the data input by the user; and the central processor 901 can be used to call the posture calculation program stored in the memory 905, and specifically perform the following operations:

[0161] Obtain first plane information in a current frame depth image corresponding to a current frame color image; if it is determined based on the first plane information that there is target feature information in the current frame color image, confirm whether there is target historical feature information that matches the target feature information; if there is target historical feature information that matches the target feature information, calculate the rotation component in the current posture of the electronic device based on the target historical feature information and the target feature information; calculate the translation component in the current posture of the electronic device, and calculate the current posture of the electronic device based on the rotation component and the translation component.

[0162] Optionally, the target historical feature information is obtained by transforming the target feature information in the historical frame color image into a preset reference system according to the rotation component of the historical posture corresponding to the historical frame color image;

[0163] Calculating the rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information, including: calculating the rotation component in the current posture of the electronic device according to the transformation relationship between the target historical feature information and the target feature information in the current frame color image.

[0164] Optionally, confirming whether there is target historical feature information that matches the target feature information includes: obtaining the plane angle relationship between the target feature information and the historical feature information after being transformed into a preset reference system; if the plane angle relationship satisfies the preset plane angle relationship, confirming that there is target historical feature information that matches the target feature information, and determining the historical feature information corresponding to the preset plane angle relationship as the target historical feature information.

[0165] Optionally, calculating the translation component in the current posture of the electronic device includes: solving the translation component in the current posture of the electronic device based on the rotation component in the current posture.

[0166] Optionally, the translation component in the current posture of the electronic device is solved based on the rotation component in the current posture, including: obtaining a current frame color image and a reference frame color image corresponding to the current frame color image, and constructing a reprojection error of feature points between the current frame color image and the reference frame color image; based on the rotation component and the reprojection error in the current posture, solving the translation component in the current posture of the electronic device.

[0167] Optionally, constructing a reprojection error of feature points between the current frame color image and the reference frame color image, including: determining a target feature point among the feature points of the reference frame color image according to the feature points of the reference frame color image and the second plane information of the reference depth image corresponding to the reference frame color image, wherein the target feature point is within a preset range of the same plane to be optimized of the reference depth image; constructing a reprojection error of feature points between the current frame color image and the reference frame color image based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device; solving a translation component in the current posture of the electronic device based on the rotation component and the reprojection error in the current posture, including: obtaining a target matching feature point corresponding to the target feature point in the current frame color image, solving a minimum value of the reprojection error based on the target matching feature point and the rotation component, and obtaining the translation component in the current posture of the electronic device.

[0168] Optionally, if there is no target historical feature information that matches the target feature information, the current frame color image and the reference frame color image corresponding to the current frame color image are obtained, and a reprojection error of the feature points between the current frame color image and the reference frame color image is constructed; based on the reprojection error, the rotation component and the translation component in the current posture of the electronic device are solved.

[0169] Optionally, constructing a reprojection error of feature points between a current frame color image and a reference frame color image, including: determining a target feature point among the feature points of the reference frame color image according to the feature points of the reference frame color image and the second plane information of the reference depth image corresponding to the reference frame color image, wherein the target feature point is within a preset range of the same plane to be optimized of the reference depth image; constructing a reprojection error of feature points between the current frame color image and the reference frame color image based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device; solving a rotation component and a translation component in the current posture of the electronic device based on the reprojection error, including: obtaining a target matching feature point corresponding to the target feature point in the current frame color image, solving a minimum value of the reprojection error based on the target matching feature point, and obtaining a rotation component and a translation component in the current posture of the electronic device.

[0170] Optionally, after solving the rotation component and the translation component in the current posture of the electronic device based on the reprojection error, it also includes: obtaining historical feature information matching the target feature information according to the target feature information of the current frame color image, and saving the historical feature information.

[0171] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0172] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0174] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0175] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0176] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0177] The above is a description of a posture calculation method, device, storage medium and electronic device provided in this application. For technical personnel in this field, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A posture calculation method, characterized in that: Applied to electronic equipment, the method comprises: Obtaining first plane information in a current frame depth image corresponding to a current frame color image; If it is determined according to the first plane information that target feature information exists in the current frame color image, then confirming whether there is target historical feature information that matches the target feature information; If there is target historical feature information that matches the target feature information, calculating a rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information; Calculating a translation component in a current posture of the electronic device, and calculating the current posture of the electronic device according to the rotation component and the translation component; If there is no target historical feature information matching the target feature information, obtaining the current frame color image and a reference frame color image corresponding to the current frame color image, and constructing a reprojection error of feature points between the current frame color image and the reference frame color image; The rotation component and the translation component in the current posture of the electronic device are solved based on the reprojection error.

2. The method according to claim 1, characterized in that The target historical feature information is obtained by transforming the target feature information in the historical frame color image into a preset reference system according to the rotation component of the historical posture corresponding to the historical frame color image; The calculating the rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information includes: The rotation component in the current posture of the electronic device is calculated according to the transformation relationship between the target historical feature information and the target feature information in the current frame color image.

3. The method according to claim 1, characterized in that The confirming whether there is target historical feature information matching the target feature information includes: Obtaining the plane angle relationship between the target feature information and the historical feature information after being transformed into a preset reference system; If the plane angle relationship satisfies the preset plane angle relationship, it is confirmed that there is target historical feature information matching the target feature information, and the historical feature information corresponding to the preset plane angle relationship is determined as the target historical feature information.

4. The method according to claim 1, characterized in that: The calculating the translation component in the current posture of the electronic device comprises: The translation component in the current posture of the electronic device is solved based on the rotation component in the current posture.

5. The method according to claim 4, characterized in that The step of solving the translation component in the current posture of the electronic device based on the rotation component in the current posture comprises: Acquire the current frame color image and a reference frame color image corresponding to the current frame color image, and construct a reprojection error of feature points between the current frame color image and the reference frame color image; Based on the rotation component in the current posture and the reprojection error, the translation component in the current posture of the electronic device is solved.

6. The method according to claim 5, characterized in that The constructing a reprojection error of feature points between the current frame color image and the reference frame color image includes: Determine, according to the feature points of the reference frame color image and the second plane information of the reference depth image corresponding to the reference frame color image, a target feature point among the feature points of the reference frame color image, wherein the target feature point is within a preset range of the same plane to be optimized of the reference depth image; Based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device, construct a reprojection error of feature points between the current frame color image and the reference frame color image; The step of solving the translation component in the current posture of the electronic device based on the rotation component in the current posture and the reprojection error includes: The target matching feature points corresponding to the target feature points in the current frame color image are obtained, and the minimum value of the reprojection error is solved based on the target matching feature points and the rotation component to obtain the translation component in the current posture of the electronic device.

7. The method according to claim 1, characterized in that The constructing a reprojection error of feature points between the current frame color image and the reference frame color image includes: Determine, according to the feature points of the reference frame color image and the second plane information of the reference depth image corresponding to the reference frame color image, a target feature point among the feature points of the reference frame color image, wherein the target feature point is within a preset range of the same plane to be optimized of the reference depth image; Based on the plane parameters to be optimized of the plane to be optimized and the posture to be optimized of the electronic device, construct a reprojection error of feature points between the current frame color image and the reference frame color image; The step of solving the rotation component and the translation component in the current posture of the electronic device based on the reprojection error includes: The target matching feature points corresponding to the target feature points in the current frame color image are obtained, and the minimum value of the reprojection error is solved based on the target matching feature points to obtain the rotation component and the translation component in the current posture of the electronic device.

8. The method according to claim 1, characterized in that After solving the rotation component and the translation component in the current posture of the electronic device based on the reprojection error, the method further includes: According to the target feature information of the current frame color image, historical feature information matching the target feature information is obtained, and the historical feature information is saved.

9. A posture calculation device, characterized in that: Applied to electronic equipment, the device comprises: A plane information acquisition module, used to acquire first plane information in a current frame depth image corresponding to a current frame color image; A historical feature information determination module, configured to, if it is determined according to the first plane information that target feature information exists in the color image of the current frame, confirm whether there is target historical feature information that matches the target feature information; A rotation component calculation module, configured to calculate the rotation component in the current posture of the electronic device according to the target historical feature information and the target feature information if there is target historical feature information matching the target feature information; A current posture calculation module, used to calculate the translation component in the current posture of the electronic device, and calculate the current posture of the electronic device according to the rotation component and the translation component; The current posture calculation module is also used to obtain the current frame color image and the reference frame color image corresponding to the current frame color image if there is no target historical feature information matching the target feature information, and construct a reprojection error of the feature points between the current frame color image and the reference frame color image; and solve the rotation component and the translation component in the current posture of the electronic device based on the reprojection error.

10. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 8 when executing the program.

Images