Camera posture determination method, device, electronic device and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for determining the position and posture of a camera, and relates to the technical field of electronic devices. A plurality of color images and a plurality of depth images obtained by shooting an environmental reference object with a camera are obtained, feature points of the environmental reference object in the color image are extracted, and the feature points of the plurality of color images are reprojected to establish a reprojection error of the plurality of color images, and the plane features of the environmental reference object in the depth image are extracted as the plane features of the corresponding environmental reference object in the color image, and the plane features of the plurality of color images are matched to establish a plane error constraint of the plurality of color images, and the position and posture of the camera are determined based on the reprojection error constraint and the plane error constraint. The present application extracts plane features through depth images, so that the plane extraction will not be affected by the imaging quality of the color image, and in dark, textureless and other environments, effective plane features can still be stably proposed, and the positioning is more robust and accurate.

Description

Technical Field

[0001] The present application relates to the technical field of electronic devices, and more specifically, to a method and device for determining the position and posture of a camera, an electronic device, and a storage medium. Background Art

[0002] Nowadays, industries such as mobile indoor navigation and 3D reconstruction, augmented reality, and mobile robots are developing rapidly and have broad application prospects and commercial value in the future. These industries or applications are inseparable from a key technology - real-time positioning technology. The current real-time positioning technology can show good tracking effects in environments with rich and highly differentiated environmental textures, but in environments with weak textures, no textures, or repeated textures, its positioning effect will be severely reduced or even directly fail, resulting in poor positioning effects. Summary of the invention

[0003] In view of the above problems, the present application proposes a method, device, electronic device and storage medium for determining the position and posture of a camera to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a method for determining the position and posture of a camera, the method comprising: acquiring multiple frames of color images and multiple frames of depth images obtained by photographing an environmental reference object by a camera; extracting feature points of the environmental reference object in the color image, and reprojecting the feature points of the multiple frames of color images to establish a reprojection error constraint for the multiple frames of color images; extracting planar features of the environmental reference object in the depth image as planar features of the environmental reference object in the corresponding color image, and matching the planar features of the multiple frames of color images to establish a planar error constraint for the multiple frames of color images; and determining the position and posture of the camera based on the reprojection error constraint and the planar error constraint.

[0005] In a second aspect, an embodiment of the present application provides a device for determining the posture of a camera, the device comprising: an image acquisition module, used to acquire multiple frames of color images and multiple frames of depth images obtained by photographing an environmental reference object with a camera; a reprojection error constraint establishment module, used to extract feature points of the environmental reference object in the color image, and reproject the feature points of the multiple frames of color images to establish reprojection error constraints for the multiple frames of color images; a plane error constraint establishment module, used to extract plane features of the environmental reference object in the depth image as plane features of the environmental reference object in the corresponding color image, and match the plane features of the multiple frames of color images to establish plane error constraints for the multiple frames of color images; a camera posture determination module, used to determine the posture of the camera based on the reprojection error constraints and the plane error constraints.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0008] The camera posture determination method, device, electronic device and storage medium provided in the embodiments of the present application obtain multiple color images and multiple depth images obtained by shooting environmental reference objects with a camera, extract feature points of the environmental reference objects in the color images, reproject the feature points of the multiple color images, establish reprojection errors of the multiple color images, extract planar features of the environmental reference objects in the depth images as planar features of the corresponding environmental reference objects in the color images, match the planar features of the multiple color images, establish planar error constraints of the multiple color images, determine the camera posture based on the reprojection error constraints and the planar error constraints, and extract planar features through the depth images. In this way, the plane extraction will not be affected by the imaging quality of the color images, and can still stably propose effective planar features in dark, texture-free environments, so that the positioning is more robust and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of a process of determining a camera's position and posture provided by an embodiment of the present application is shown;

[0011] Figure 2 A schematic diagram of a process of determining a camera's position and posture provided by an embodiment of the present application is shown;

[0012] Figure 3 A schematic diagram of a color image provided by an embodiment of the present application is shown;

[0013] Figure 4 A schematic diagram of a process of determining a camera's position and posture provided by an embodiment of the present application is shown;

[0014] Figure 5 A schematic diagram showing the comparison of two color images provided in an embodiment of the present application is shown;

[0015] Figure 6 This application shows Figure 4 A schematic flow chart of step S305 of the method for determining the position and posture of a camera shown;

[0016] Figure 7 A module block diagram of a camera posture determination device provided in an embodiment of the present application is shown;

[0017] Figure 8 A block diagram of an electronic device for executing a method for determining a camera's posture according to an embodiment of the present application is shown;

[0018] Fig. 9 A storage unit for storing or carrying program codes for implementing a method for determining a camera's posture according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0020] Nowadays, industries such as mobile indoor navigation and 3D reconstruction, augmented reality, and mobile robots are developing rapidly and have broad application prospects and commercial value in the future. These industries or applications are inseparable from a key technology - real-time positioning technology. Among them, electronic devices (such as sweeping robots) need centimeter-level self-positioning to know the position relationship between themselves and the environment, so as to carry out subsequent tasks. Obviously, the global positioning system (GPS) cannot meet the requirements of positioning accuracy. Simultaneous localization and mapping (SLAM) and visual-inertial navigation fusion odometer (Visual-IMU odometry, VIO) technologies have emerged as the times require. These technologies refer to the use of information provided by the electronic device's own sensors in a completely unknown environment, combined with corresponding algorithms to achieve centimeter-level positioning. At the same time, SLAM technology also includes environmental mapping while positioning. On electronic devices such as mobile phones or mobile robots, the most commonly installed sensor is the camera sensor. The positioning process of the camera body is completed through the image sequence collected by the camera, which is called visual SLAM technology.

[0021] In the classic visual SLAM algorithm, the relative position between cameras is solved by extracting feature points from the image, establishing matching relationships between image feature points under different perspectives, and then using multi-view geometric constraints.

[0022] In 2015, an open-source positioning solution ORB-SLAM based on a rotationally invariant visual feature (oriented fast and rotated brief, ORB) was proposed. It provides a complete set of real-time and high-precision feature point visual SLAM solutions, becoming a benchmark work in the SLAM academic community. ORB feature points are extracted quickly and are scale and rotationally invariant, establishing associations and constraints between continuous image frames in the entire positioning system. The entire ORB-SLAM system has established a rigorous algorithm flow around the ORB feature, including initialization, pose prediction, tracking local maps, key frame screening, local BA optimization, loop detection and other modules.

[0023] In 2017, VINS-MONO and VINS-MOBILE, two VIO positioning solutions, were proposed, which provided a complete monocular camera and IMU tightly coupled positioning solution that can be used on a computer or mobile device. The visual positioning module of VINS relies on the visual feature points Shi-Tomasi corner points. First, the Shi-Tomasi corner points are extracted from the continuous image frames passed into the system, and then the corner points are tracked using LK optical flow to achieve feature association between image frames, and then visual constraints are established to calculate the pose of the image frames.

[0024] Since 2010, with the continuous maturity of depth camera technology, many SLAM scholars have begun to use RGB cameras combined with depth cameras for visual positioning, hereinafter referred to as RGBD-SLAM. In 2019, someone expanded the VINS-MONO system, integrated depth camera-assisted positioning, and proposed and open-sourced a positioning solution using a depth camera and IMU for tight coupling. Referred to as RGBD-VINS. In the positioning initialization and tracking stages, the depth information provided by the depth camera is used to restore the two-dimensional feature points to three-dimensional space points, making the VINS positioning system initialize faster and improving tracking accuracy.

[0025] Using image feature points for positioning can establish a stable correspondence between images, but its shortcomings are also obvious. In some scenes with weak textures, such as dark environments and white wall environments, the images collected in these environments lack grayscale gradients, making it difficult to extract feature points from them, which in turn leads to subsequent positioning failures. In some repetitive texture environments, such as striped wall environments, although a lot of environmental feature points can be extracted, the extracted feature points are very similar and lack distinction, which can easily cause feature mismatching, resulting in incorrect matching between images, large positioning errors, or even failure.

[0026] Compared with the classic traditional visual SLAM algorithm, it can show better tracking effect in the environment with rich texture and high differentiation, but its positioning effect will be seriously reduced or even fail directly in the environment with weak texture, no texture, and repeated texture. This is because it is impossible to propose enough feature points and establish the correct constraint relationship in these environments. This has also become a weakness of the current feature point method visual SLAM.

[0027] In view of the above problems, the inventors have discovered after long-term research and proposed the camera posture determination method, device, electronic device and storage medium provided in the embodiments of the present application. The plane features are extracted by the depth image, so that the plane extraction will not be affected by the imaging quality of the color image. In the dark, textureless environment, it is still possible to stably propose effective plane features, and the robustness is stronger. Among them, the specific camera posture determination method is described in detail in the subsequent embodiments.

[0028] See also Figure 1 , Figure 1 The flowchart of the method for determining the position and posture of a camera provided by an embodiment of the present application is shown. The method for determining the position and posture of a camera is used to extract plane features through a depth image, so that the plane extraction will not be affected by the imaging quality of the color image, and in a dark, textureless environment, effective plane features can still be stably extracted, and the positioning is more robust and accurate. In a specific embodiment, the method for determining the position and posture of a camera is applied to Figure 7 The camera posture determination device 200 and the electronic device 100 equipped with the camera posture determination device 200 are shown in FIG. Figure 8 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment may include a smart phone, a tablet computer, a robot, etc., which is not limited here. Figure 1 The process shown is described in detail, and the method for determining the position and posture of the camera may specifically include the following steps:

[0029] Step S101: acquiring multiple frames of color images and multiple frames of depth images obtained by photographing environmental reference objects with a camera.

[0030] In this embodiment, the environmental reference object is an environmental entity in the camera shooting direction, such as the ground, walls, furniture, etc. The environmental reference object will change continuously with the movement of the camera, depending on the shooting direction of the camera. In this embodiment, the camera can be set on a mobile device for moving and shooting environmental reference objects. For example, the camera can be set on a mobile robot (such as a sweeping robot, a mopping robot, a serving robot, etc.), and shoot the environmental reference object as the mobile robot moves.

[0031] In some embodiments, the camera may have a depth sensor and a color sensor (RGB sensor). The environmental reference objects photographed by the depth sensor and the RGB sensor are the same, and the depth sensor and the RGB sensor photograph the same environmental reference object synchronously, that is, the shooting parameters such as shooting time, shooting distance, shooting angle, etc. of the depth sensor and the RGB sensor for photographing the environmental reference object should be the same. The color image is the image data obtained by photographing the environmental reference object based on the RGB sensor, and the depth image is the image data obtained by photographing the environmental reference object based on the depth sensor. Exemplarily, the camera in this embodiment may be an RGB-D camera.

[0032] In some embodiments, during the process of photographing the environmental reference object by the camera, the camera may be controlled to continuously photograph the environmental reference object to obtain multiple color image frames and multiple depth image frames, wherein the multiple color image frames and the multiple depth image frames correspond one to one. For example, during the process of the camera moving, the camera may be controlled to continuously photograph the environmental reference object to obtain multiple color image frames and multiple depth image frames.

[0033] In some embodiments, when a color image and a depth image are obtained, it is possible to detect whether the color image and the depth image are aligned in time and space. Among them, when it is detected that the color image and the depth image are not aligned in time and space, the color image and the depth image can be first aligned in time and space, and then subsequently input into the SLAM algorithm. As an implementable method, it is possible to detect whether the camera has a built-in alignment function. If the camera has a built-in alignment function, it can be assumed that the color image and the depth image are aligned in time and space, and no longer detect whether the color image and the depth image are aligned in time and space; if the camera does not have a built-in alignment function, it can be assumed that it is necessary to detect whether the color image and the depth image are aligned in time and space.

[0034] Step S102: extracting feature points of the environmental reference object in the color image, and reprojecting the feature points of the multiple frames of color images to establish reprojection error constraints of the multiple frames of color images.

[0035] In this embodiment, when multiple frames of color images are obtained, feature points of environmental reference objects in each frame of the multiple frames of color images can be extracted, and the feature points of the multiple frames of color images can be reprojected to establish reprojection error constraints for the multiple frames of color images.

[0036] In some embodiments, when multiple color images are obtained, one of the multiple color images can be distorted and corrected, and the fast corner points in the one of the color images can be extracted as feature points. Then, the fast corner points on other color images in the multiple color images can be tracked and matched using the optical flow tracking method. Since the depth image is obtained at the same time as the color image, the depth value of each feature point on the color image can be obtained based on the corresponding depth image, so that the feature points with abnormal depth or no depth in the color image (finding feature points whose depth value is 0 on the depth image or feature points that exceed the range of the depth camera (greater than 6m)) can be removed, and the remaining feature points with good depth can be back-projected into space to form three-dimensional map points. These three-dimensional map points are projected onto other common view frames, and the reprojection error constraints of the multiple color images can be established.

[0037] It should be noted that many feature points can be seen on color images of different viewing angles, and these feature points can be matched by the optical flow method, that is, the matching between 2D feature points and 2D feature points under different viewing angles can be obtained. Then, the 2D feature points are back-projected into 3D map points according to the depth information, so that the matching between the 3D feature points of the current frame and the 2D feature points of other frames can be formed. If the accurate posture transformation relationship between the current frame and other frames is determined, when the 3D feature points of the current frame are projected onto other frames, the projected 2D feature points and the matched 2D feature points should completely overlap. However, in fact, due to the inaccurate posture transformation relationship between the current frame and other frames, when the 3D feature points of the current frame are projected onto other frames, the projected 2D feature points and the matched 2D feature points do not overlap, thus forming a reprojection error. Among them, the posture transformation relationship can be optimized by minimizing the reprojection error to make the posture transformation relationship more accurate.

[0038] In some embodiments, in order to ensure the real-time performance of the SLAM system, the optimization scale can be limited. For example, key frame color images can be selected from multiple frames of color images to constrain the color images to ensure the optimization speed. For another example, a color image in a window can be selected from multiple frames of color images in a sliding window manner to constrain the color images to ensure the optimization speed. Optionally, the window size of the sliding window can be set to 10 frames, and then only the 10 frames of color images in the sliding window can be constrained.

[0039] Step S103: extracting the plane features of the environmental reference object in the depth image as the plane features of the environmental reference object in the corresponding color image, and matching the plane features of the multiple frames of color images to establish plane error constraints of the multiple frames of color images.

[0040] Among them, step S102 can be executed before step S103, can be executed at the same time as step S103, or can be executed after step S103, which is not limited here.

[0041] In this embodiment, when multiple frames of depth images are obtained, the plane features of the environmental reference object in each frame of the multiple frames of depth images can be extracted as the plane features of the environmental reference object in the corresponding color image, and the plane features of the multiple frames of color images are matched to establish the plane error constraints of the multiple frames of color images. It can be understood that since the color image and the depth image with a corresponding relationship are aligned in time and space, the plane features of the environmental reference object collected on the depth image should also be consistent with the plane features of the environmental reference object on the color image with a corresponding relationship, that is, the plane features of the environmental reference object extracted from the depth image can be used as the plane features of the environmental reference object in the corresponding color image.

[0042] In some embodiments, when multiple frames of depth images are obtained, plane fitting can be performed on each frame of the multiple depth images, so as to extract the planar features of the environmental reference object from each frame of the depth image as the planar features of the environmental reference object in the corresponding color image by plane fitting. Then, the planar features of the multiple frames of color images are matched, and the planar error constraints of the multiple frames of color images can be established.

[0043] In some embodiments, when multiple frames of depth images are obtained, each frame of the multiple frames of depth images is divided into multiple basic image blocks, and a basic image block is selected from the multiple basic image blocks as a target image block, and the target image block is fused with the basic image block adjacent to it to obtain a fused image block, and a plane fitting is performed on the fused image block to obtain an average error of the plane fitting as a first error. Based on the relationship between the first error and the error threshold, the multiple basic image blocks are processed until the multiple basic image blocks are processed. The plane features of the environmental reference object in the processed depth image are obtained as the plane features of the environmental reference object in the corresponding color image, and then the plane features of the multiple frames of color images are matched to establish a plane error constraint for the multiple frames of color images.

[0044] In some embodiments, matching planar features of multiple color images and establishing planar error constraints for the multiple color images may include: matching identical planar features in multiple color images to establish planar error constraints for the multiple color images; matching parallel planar features in multiple color images to establish planar error constraints for the multiple color images; matching vertical planar features in multiple color images to establish planar error constraints for the multiple color images; matching identical planar features and parallel planar features in multiple color images to establish planar error constraints for the multiple color images; matching identical planar features and vertical planar features in multiple color images to establish planar error constraints for the multiple color images; matching parallel planar features and vertical planar features in multiple color images to establish planar error constraints for the multiple color images; matching identical planar features, parallel planar features, and vertical planar features in multiple color images to establish planar error constraints for the multiple color images.

[0045] It can be understood that in the process of extracting plane features, this embodiment uses a depth image instead of relying on a color image to extract plane features. In this way, the extracted plane features will not be affected by the imaging quality of the color image, and effective plane features can still be stably extracted in dark, textureless and other environments.

[0046] Step S104: Determine the position and posture of the camera based on the reprojection error constraint and the plane error constraint.

[0047] In this embodiment, when the reprojection error constraints corresponding to the multiple frames of color images and the plane error constraints corresponding to the multiple frames of color images are obtained, the position and posture of the camera can be determined based on the reprojection error constraints and the plane error constraints.

[0048] In some embodiments, when the reprojection error constraints corresponding to multiple frames of color images and the plane error constraints corresponding to the multiple frames of color images are obtained, an error function of tightly coupled multi-constrained errors can be established based on the reprojection error constraints and the plane error constraints. After constructing the error function of tightly coupled multi-constrained errors, a nonlinear optimization method can be used to optimize the error function of tightly coupled multi-constrained errors to solve the optimal image frame pose, that is, the pose of the camera.

[0049] A method for determining the position and posture of a camera provided in an embodiment of the present application obtains multiple color images and multiple depth images obtained by shooting an environmental reference object with a camera, extracts feature points of the environmental reference object in the color image, reprojects the feature points of the multiple color images, establishes a reprojection error of the multiple color images, extracts planar features of the environmental reference object in the depth image as planar features of the corresponding environmental reference object in the color image, matches the planar features of the multiple color images, establishes planar error constraints of the multiple color images, determines the position and posture of the camera based on the reprojection error constraints and the planar error constraints, and extracts planar features through the depth image. In this way, the plane extraction will not be affected by the imaging quality of the color image, and can still stably propose effective planar features in dark, texture-free environments, and has stronger positioning robustness and higher accuracy.

[0050] See also Figure 2 , Figure 2 The flowchart of the method for determining the position and posture of a camera provided in one embodiment of the present application is shown in FIG. Figure 2 The process shown is described in detail, and the method for determining the position and posture of the camera may specifically include the following steps:

[0051] Step S201: Acquire multiple frames of color images and multiple frames of depth images obtained by photographing environmental reference objects with a camera.

[0052] Step S202: extracting feature points of the environmental reference object in the color image, and reprojecting the feature points of the multiple frames of color images to establish reprojection error constraints of the multiple frames of color images.

[0053] For the detailed description of step S201 - step S202 , please refer to step S101 - step S102 , which will not be repeated here.

[0054] Step S203: Divide the depth image into a plurality of basic image blocks.

[0055] In this embodiment, when a depth image is obtained, the depth image can be divided into a plurality of basic image blocks. For each depth image in the plurality of depth images, each depth image can be divided into a plurality of basic image blocks in the above manner, which will not be described in detail here.

[0056] In some embodiments, a preset size may be pre-set and stored, and when a depth image is obtained, the depth image may be divided into a plurality of basic image blocks according to the preset size. As an implementable manner, the preset size may include 10*10, and based on the size of 10*10, the speed and accuracy of plane fitting for the depth image may be balanced.

[0057] Step S204: selecting a basic image block from the multiple basic image blocks as a target image block, and fusing the target image block with adjacent basic image blocks to obtain a fused image block.

[0058] In this embodiment, when the depth image is divided into multiple basic image blocks, a basic image block can be selected from the multiple basic image blocks as a target image block, and the target image block and the basic image blocks adjacent to it are fused, and the image block obtained by fusion is used as a fused image block.

[0059] In some embodiments, a basic image block can be randomly selected from multiple basic image blocks as the target image block, a basic image block located in the middle can be selected from multiple basic image blocks as the target image block, or a basic image block can be selected from basic image blocks located in corners of multiple image blocks as the target image block, etc., which are not limited here. As an implementable manner, if the target image block is adjacent to four basic image blocks, the target image block can be fused with the four basic image blocks in sequence to obtain a fused image block; if the target image block is adjacent to two basic image blocks, the target image block can be fused with the two basic image blocks in sequence to obtain a fused image block.

[0060] In some embodiments, a first preset condition may be pre-set and stored, and the first preset condition may be used as a basis for determining the basic image block. Therefore, in this embodiment, when multiple basic image blocks are obtained, the basic image blocks that meet the first preset condition may be filtered from the multiple basic image blocks, and the remaining basic image blocks from the multiple basic image blocks are retained as image blocks to be processed, and then one image block to be processed is selected from the multiple image blocks to be processed as a target image block, and the target image block and the adjacent image blocks to be processed are fused, and the image block obtained by fusion is used as a fused image block.

[0061] The first preset condition may be a condition that characterizes an image block as an unqualified image block. Therefore, if the basic image block satisfies the first preset condition, the basic image block may be considered to be an unqualified image block, and the basic image block may be filtered out from multiple basic image blocks; if the basic image block does not satisfy the first preset condition, the basic image block may be considered to be a qualified image block, and the basic image block may be retained from multiple basic image blocks as an image block to be processed.

[0062] As a method, the first preset condition may include the existence of pixels containing invalid depth information. That is, when multiple basic image blocks are obtained, for each basic image block in the multiple basic image blocks, the depth information of each pixel contained in the basic image block is detected to determine whether there are pixels containing invalid depth information in the basic image block. Wherein, when it is determined that there are pixels containing invalid depth information in the basic image block, it can be determined that the basic image block meets the first preset condition, and the basic image block can be filtered out from the multiple basic image blocks. Wherein, the pixel containing invalid depth information refers to a pixel with a depth value of 0 or a pixel that exceeds the range of the depth camera. Based on the above method, the basic image blocks with good depth information can be screened out to participate in subsequent image fusion, thereby improving the effect of image fusion.

[0063] As another way, the first preset condition may include adjacent pixels with sudden changes in depth information. That is, when multiple basic image blocks are obtained, for each basic image block in the multiple basic image blocks, the depth information of each pixel contained in the basic image block is detected to determine whether there is a sudden change in the depth information of the adjacent pixel. Wherein, when it is determined that there are adjacent pixels with sudden changes in depth information in the basic image block, it can be determined that the basic image block meets the first preset condition, and the basic image block can be filtered out from the multiple basic image blocks. Wherein, the adjacent pixels with sudden changes in depth information refer to the difference in depth information being greater than the preset difference. Wherein, it can be understood that the depth information of the pixels on the same plane is generally the same or the difference is very small. If there are adjacent pixels with sudden changes in depth information in the basic image block, it indicates that the pixels contained in the basic image block do not belong to the same plane, and the basic image block can be filtered out, thereby screening out the basic image blocks containing pixels belonging to the same plane to participate in subsequent image fusion, thereby improving the effect of image fusion.

[0064] As another way, the first preset condition may include that the average error of the plane fitting is greater than the error threshold. That is, in the case of obtaining multiple basic image blocks, for each basic image block in the multiple basic image blocks, a plane fitting is performed in the block of the basic image block to obtain the average error of the plane fitting, and then the average error is compared with the preset error threshold to determine whether the average error is greater than the error threshold. Wherein, in the case of determining that the average error of the plane fitting of the basic image block is greater than the error threshold, it can be determined that the basic image block meets the first preset condition, and the basic image block can be filtered out from the multiple basic image blocks. Wherein, performing plane fitting in the block of the basic image block will fit the plane parameters n (normal vector) and d (distance) representing the plane model, and then, the 3D space point P corresponding to each pixel point in the basic image block is brought into the plane model nP+d=0, and the degree of deviation of the result from 0 reflects the quality of the plane fitting, and all errors are averaged to obtain the average error of the plane fitting. Among them, it can be understood that if most of the pixels on the basic image block belong to different planes, the average error of the plane fitting is greater than the error threshold, which indicates that the pixels contained in the basic image block do not belong to the same plane. In this case, the basic image block can be filtered out, thereby selecting the basic image blocks whose pixels basically belong to the same plane to participate in subsequent image fusion, thereby improving the effect of image fusion.

[0065] In some embodiments, when multiple basic image blocks are obtained, a basic image block can be selected from the multiple basic image blocks as the target image block, and the angle between the normal vector of the target image block and the normal vector of the adjacent basic image block is obtained. When the angle is less than the angle threshold, the target image block and the adjacent basic image block are fused, and the fused image block is used as the fused image block. Thus, through the two-stage plane matching method, the plane under different viewing angles is initially verified using the normal vector direction, and then the subsequent error verification is performed, which can achieve faster matching of plane features.

[0066] As an implementable manner, an angle threshold value may be pre-set and stored, and the angle threshold value is used as a basis for determining the angle between normal vectors of adjacent basic image blocks. Therefore, in this embodiment, when multiple basic image blocks are obtained, one basic image block may be selected from the multiple basic image blocks as a target image block, and a basic image block adjacent to the target image block may be determined. Then, the target image block is subjected to plane fitting within the block to fit the normal vector of the target image block, and the basic image blocks adjacent to the target image block are subjected to plane fitting within the block to fit the normal vectors of the basic image blocks adjacent to the target image block, respectively. When the normal vector of the target image block and the normal vector of the basic image blocks adjacent to the target image block are obtained, the angle between the normal vector of the target image block and the normal vector of the basic image blocks adjacent to the target image block may be calculated. Afterwards, the angle is compared with the angle threshold to determine whether the angle is less than the angle threshold. When it is determined that the angle is less than the angle threshold, the normal vector representing the target image block and the normal vector of the basic image block adjacent to the target image block are not much different, then the target image block and the basic image block adjacent to it can be fused. When it is determined that the angle is greater than or equal to the angle threshold, the normal vector representing the target image block and the normal vector of the basic image block adjacent to the target image block are significantly different, then the target image block and the basic image block adjacent to it may not be fused.

[0067] Step S205: performing plane fitting on the fused image block, and obtaining an average error of the plane fitting as a first error.

[0068] In this embodiment, when a fused image block is obtained, a plane fitting may be performed on the fused image block, and an average error of the plane fitting may be obtained as the first error. In some implementations, when a fused image block is obtained, a plane fitting may be performed within the fused image block, and plane parameters n (normal vector) and d (distance) representing the plane model may be fitted. Then, the 3D space point P corresponding to each pixel point in the fused image block is brought into the plane model nP+d=0, and the degree of deviation of the result from 0 reflects the quality of the plane fitting. All errors are averaged, and the average error of the plane fitting may be obtained as the first error.

[0069] Step S206: Based on the relationship between the first error and the error threshold, the multiple basic image blocks are processed until the multiple basic image blocks are processed, and then the plane features of the environmental reference object in the depth image obtained by processing are obtained.

[0070] In some implementations, an error threshold may be pre-set and stored, and the error threshold is used as a basis for judging the average error of the plane fitting of the fused image block. Therefore, in this embodiment, when the first error is obtained, the first error may be compared with the error threshold to obtain the relationship between the first error and the error threshold. When the relationship between the first error and the error threshold is obtained, multiple basic image blocks may be processed based on the relationship between the first error and the error threshold until the multiple basic image blocks are processed, and the plane features of the environmental reference objects in the depth image obtained by processing are obtained.

[0071] As an implementable manner, the relationship between the first error and the error threshold may include that the first error is less than the error threshold, or that the first error is greater than or equal to the error threshold, and processing multiple basic image blocks may include a first processing method and a second processing method. Then, when the first error is less than the error threshold, the multiple basic image blocks may be processed by the first processing method until the multiple basic image blocks are processed, and the planar features of the environmental reference objects in the processed depth image are obtained. When the first error is greater than or equal to the error threshold, the multiple basic image blocks may be processed by the second processing method until the multiple basic image blocks are processed, and the planar features of the environmental reference objects in the processed depth image are obtained.

[0072] In some embodiments, when the first error is less than the error threshold, the fused image block is used as the target image block, and the step of fusing the target image block with the adjacent basic image blocks is repeated until multiple basic image blocks have been fused, and the fused image block is determined as the planar feature of the environmental reference object in the depth image.

[0073] Wherein, when the first error is less than the error threshold, it can be considered that the target image block and the adjacent basic image block are successfully fused, that is, the fused image block obtained by fusion belongs to a plane feature. At this time, the fused image block and the adjacent basic image block can be fused. If the error of the plane fitting of the image block obtained by the second fusion is less than the error threshold, indicating that the image block obtained by the second fusion belongs to a plane feature, the image block obtained by the second fusion is fused with the adjacent basic image block, and so on, until multiple basic image blocks have been fused, it is determined that the plane segmentation in the depth image is completed. If the error of the plane fitting of the image block obtained by the second fusion is greater than or equal to the error threshold, indicating that the image block obtained by the second fusion does not belong to a plane feature, the fused image block is determined as a plane feature of the environmental reference object, and the target image block is reselected from the remaining basic image blocks, and the above steps are repeated to obtain other plane features of the environmental reference object, until multiple basic image blocks have been fused, it is determined that the plane segmentation in the depth image is completed.

[0074] In some embodiments, when the first error is greater than or equal to an error threshold, a basic image block is reselected from multiple basic image blocks as a target image block, and the step of fusing the target image block with the basic image blocks adjacent to it is repeated until multiple basic image blocks have been fused, and the fused image block is determined as the planar feature of the environmental reference object in the depth image.

[0075] Among them, when the first error is greater than or equal to the error threshold, it can be considered that the fusion of the target image block and the adjacent basic image block is unsuccessful (failed), that is, the fused image obtained by fusion does not belong to a plane feature. At this time, the target image block can be determined as a plane feature of the environmental reference object, and the target image block is reselected from the remaining basic image blocks, and the above steps are repeated to obtain other plane features of the environmental reference object. Until multiple basic image blocks have been fused, it is determined that the plane segmentation in the depth image is completed.

[0076] Based on this, in the plane feature extraction algorithm of this embodiment, a method of fitting the plane in blocks and then continuously searching the neighborhood is adopted. This method avoids fitting the plane with many pixels together and can improve the speed of plane fitting.

[0077] See also Figure 3 , Figure 3 FIG. 1 shows a schematic diagram of a depth image provided by an embodiment of the present application. Figure 3As shown, the depth image is divided into 16 basic image blocks, namely basic image block A1, basic image block A2, basic image block A3, basic image block A4, basic image block A5, basic image block A6, basic image block A7, basic image block A8, basic image block A9, basic image block A10, basic image block A11, basic image block A12, basic image block A13, basic image block A14, basic image block A15 and basic image block A16.

[0078] If basic image block A7 is selected as the target image block from basic image blocks A1 to A16, basic image block A7 can be fused with adjacent basic image blocks A3, A6, A8, and A11. If the average error of the plane fitting of the fused image block obtained by fusion is less than the error threshold, the fused image block can be fused with adjacent basic image blocks A2, A4, A5, A10, A12, and A15, and so on. If the average error of the plane fitting of the fused image block obtained by fusion is greater than or equal to the error threshold, the target image block can be reselected from other basic image blocks except basic image block A7. For example, if basic image block A16 is selected as the target image block, basic image block A16 can be fused with adjacent basic image blocks A12 and A15, and so on. In the above manner, when all 16 basic image blocks have been fused, the plane segmentation in the depth image can be completed according to the result of the fusion judgment.

[0079] Among them, after completing the plane segmentation in the depth image, multiple initial plane features of the environmental reference object in the depth image may be obtained. In order to improve the accuracy of the determined camera posture and reduce process losses, multiple initial plane features can be screened to ensure that the screened plane features have a large area and a small error. Finally, this embodiment can quickly extract relatively accurate plane features to improve the processing accuracy of subsequent algorithms. In some embodiments, multiple initial plane features of the environmental reference object in the depth image obtained by processing can be obtained, and then the initial plane features that meet the second preset condition are filtered from the multiple initial plane features to obtain better plane features as the plane features of the environmental reference object in the depth image.

[0080] As an implementable manner, a second preset condition may be pre-set and stored, and the second preset condition may be used as a basis for determining the initial plane feature. Therefore, in this embodiment, when multiple initial plane features of the environmental reference object in the depth image are obtained, the initial plane features that meet the second preset condition may be filtered from the multiple initial plane features, and the remaining initial plane features in the multiple initial plane features are retained as the plane features of the environmental reference object in the depth image.

[0081] Among them, the second preset condition can be a condition that characterizes the plane feature as an unqualified plane feature. Therefore, when the initial plane feature meets the second preset condition, the initial plane feature block can be considered as an unqualified plane feature, and the initial plane feature can be filtered out from multiple initial plane features; when the initial plane feature does not meet the second preset condition, the initial plane feature can be considered as a qualified plane feature, and the initial plane feature can be retained from multiple initial plane features as a plane feature of the environmental reference object in the depth image.

[0082] As a method, the second preset condition may include: the number of pixels included is less than a quantity threshold. That is, when multiple initial plane features are obtained, for each of the multiple initial plane features, the number of pixels included in the initial plane feature is detected to determine whether the number of pixels included in the initial plane feature is less than the quantity threshold. Wherein, when it is determined that the number of pixels included in the initial plane feature is less than the quantity threshold, it can be considered that the area of ​​the initial plane feature is too small and unstable, and is not helpful for subsequent algorithms, then it can be determined that the initial plane feature meets the second preset condition, and then the initial plane feature can be filtered out from the multiple initial plane features. Wherein, the quantity threshold may include 10000, for example.

[0083] As another way, the second preset condition may include: the average distance from the pixels included to the corresponding initial plane feature is greater than the distance threshold. That is, in the case of obtaining multiple initial plane features, for each of the multiple initial plane features, the distance from each pixel included in the initial plane feature to the corresponding initial plane feature is detected, and in the case of detecting the distance from each pixel included in the initial plane feature to the corresponding initial plane feature, based on the distance from each pixel included in the initial plane feature to the corresponding initial plane feature, the average distance from the pixel included in the initial plane feature to the corresponding initial plane feature is calculated, and the average distance is compared with the distance threshold to determine whether the average distance is greater than the distance threshold, wherein, in the case of determining that the average distance is greater than the distance threshold, it can be considered that these pixels may not belong to a plane feature, and then it can be determined that the initial plane feature meets the second preset condition, and then the initial plane feature can be filtered out from the multiple initial plane features. Wherein, the distance threshold may include 0.01m, for example.

[0084] In some embodiments, the pixels included in the initial plane feature can be downsampled, and only one pixel is retained in a 9*9 neighborhood, so as to reduce the amount of judgment by reducing the number of pixels. Then, the average distance from the pixels retained after downsampling to the corresponding initial plane feature is counted. When it is determined that the average distance is greater than the distance threshold, it can be considered that these pixels may not belong to a plane feature, and it can be determined that the initial plane feature meets the second preset condition, and the initial plane feature can be filtered out from multiple initial plane features. Among them, the pixel retained after downsampling can be a pixel located in the middle position.

[0085] Based on the above method, plane features that meet the accuracy conditions can be extracted for optimization of the SLAM system. At the same time, the speed of plane feature detection has been measured and meets the real-time requirements of the SLAM system, which can reach 30 frames / s. For the final result of plane feature detection, the detected plane parameters and all pixel points belonging to the plane feature can be saved. The plane parameter model is: Ax+By+Cz+D=0, where the vector (A, B, C) is the plane normal vector, its modulus is 1, D is the distance from the origin to the plane, and the vector (x, y, z) is the point on the plane.

[0086] Step S207: Acquire pixel points located at the edge position of the plane feature.

[0087] In the case of obtaining a plane feature by segmentation, for the pixel points near the edge of the obtained plane feature (for example, the 10*10 neighborhood of the edge point), the image block is no longer used for plane fusion. Instead, the plane fitting error can be verified pixel by pixel. If the plane fitting error is less than the error threshold, it is determined that the pixel belongs to the plane feature. In this way, the edge of the segmented plane feature can be refined.

[0088] In this embodiment, when a plane feature is obtained, the pixel points located at the edge position of the plane feature may be obtained. For example, the pixel points located within a 10*10 neighborhood of the edge point of the plane feature may be obtained.

[0089] Step S208: performing plane fitting on the pixel point and the plane feature, and obtaining a plane fitting error as a second error.

[0090] In this embodiment, when a pixel point is obtained, plane fitting can be performed on the pixel point and the plane feature to obtain the plane fitting error corresponding to the pixel point, and the plane fitting error corresponding to the pixel point is used as the second error.

[0091] Step S209: when the second error is smaller than the error threshold, determining that the pixel point belongs to the plane feature.

[0092] In some embodiments, the pre-set and stored error threshold can also be used as a basis for judging the error of the plane fitting between the pixel point and the plane feature. Therefore, in this embodiment, when the second error is obtained, the second error can be compared with the error threshold to determine whether the second error is less than the error threshold. Wherein, when it is determined that the second error is less than the error threshold, it can be determined that the pixel point belongs to the plane feature, and when it is determined that the second error is greater than or equal to the error threshold, it can be determined that the pixel point does not belong to the plane feature.

[0093] Step S210: matching the plane features of the multiple frames of color images to establish plane error constraints of the multiple frames of color images.

[0094] Step S211: Determine the position and posture of the camera based on the reprojection error constraint and the plane error constraint.

[0095] For the detailed description of step S210 - step S211 , please refer to step S103 - step S104 , which will not be repeated here.

[0096] Compared with the method for determining the position and posture of a camera provided in an embodiment of the present application, Figure 1The camera posture determination method shown in the figure, this embodiment also adopts a block-by-block plane fitting method, and then continuously searches in the field. This method avoids fitting the plane with multiple pixels together, which can improve the speed of plane fitting. At the same time, this embodiment also sets a refined plane edge refinement process, so that the extracted plane edge is more accurate.

[0097] See also Figure 4 , Figure 4 The flowchart of the method for determining the position and posture of a camera provided in one embodiment of the present application is shown in FIG. Figure 4 The process shown is described in detail, and the method for determining the position and posture of the camera may specifically include the following steps:

[0098] Step S301: Acquire multiple frames of color images and multiple frames of depth images obtained by photographing environmental reference objects with a camera.

[0099] Step S302: extracting feature points of the environmental reference object in the color image, and reprojecting the feature points of the multiple frames of color images to establish reprojection error constraints of the multiple frames of color images.

[0100] Step S303: extracting the plane features of the environmental reference object in the depth image as the plane features of the environmental reference object in the corresponding color image.

[0101] For the detailed description of steps S301 to S303 , please refer to steps S101 to S103 , which will not be repeated here.

[0102] Step S304: Acquire the same plane features among the multiple frames of color images.

[0103] In this embodiment, when the plane features included in each of the multiple color image frames are obtained, the plane features included in each of the multiple color image frames can be matched to obtain the same plane features between the multiple color image frames.

[0104] In some embodiments, taking the case where a plurality of color image frames include a first color image frame and a second color image frame as an example, assuming that the first color image frame includes a first plane feature, and the second color image frame includes a second plane feature. Then, based on the posture conversion relationship between the first color image frame and the second color image frame, the normal vector of the first plane feature can be converted from the first coordinate system to the second coordinate system, wherein the first coordinate system corresponds to the first color image frame, and the second coordinate system corresponds to the second color image frame. When the angle between the normal vector of the first plane feature and the normal vector of the second plane feature in the second coordinate system is less than an angle threshold, based on the posture conversion relationship between the first color image frame and the second color image frame, the pixel points of the first plane feature are converted from the first coordinate system to the second coordinate system. When the average distance between the pixel points of the first plane feature and the pixel points of the second plane feature in the second coordinate system is less than a distance error threshold, it can be determined that the first plane feature and the second plane feature are the same plane features.

[0105] See also Figure 5 , Figure 5 FIG. 2 shows a schematic diagram of comparing two color images provided in an embodiment of the present application. Figure 5 As shown, it is assumed that the two color image frames include a first color image I 1 and the second color image I 2 , and the first frame color image I 1 and the second color image I 2 The rotation transformation relationship between them is R ji , the translation transformation relationship is t ji , the first frame color image I 1 The first plane feature PL 1 , its plane parameters are (a 1 ,b 1 ,c 1 ,d 1 ), where vector n 1 (a 1 ,b 1 ,c 1 ) is the normal vector of the first plane feature, d 1 is the distance from the origin to the first plane feature; the second frame color image I 2 The second plane feature PL 2 , its plane parameters are (a 2 ,b 2 ,c 2 ,d 2 ), where vector n 2 (a 2 ,b 2 ,c 2 ) is the normal vector of the second plane feature, d 2is the distance from the origin to the second plane feature.

[0106] Based on this, the first plane feature PL 1 The normal vector n 1 Through the first frame color image I 1 and the second color image I 2 The pose transformation relationship between the first frame color image I 1 The corresponding first coordinate system is converted to the second frame color image I 2 In the corresponding second coordinate system, it is recorded as n' 2 .

[0107] n' 2 =R ji *n 1

[0108] After that, judge n' 2 and n 2 Is the angle between them less than an angle threshold, where the angle threshold may include 5°. 2 and n 2 If the absolute value of the angle between is greater than or equal to the angle threshold, it is considered that the first plane feature PL 1 and the second plane feature PL 2 If they do not belong to the same plane feature, the next step can be skipped. The calculation formula of the angle θ is as follows, where · represents the dot multiplication operation of the vector:

[0109]

[0110] Among them, if n' 2 and n 2 The absolute value of the angle between is less than the angle threshold, and further judgment is made, then the first plane feature PL 1 The pixel points of the first frame color image I 1 and the second color image I 2 The pose transformation relationship between the first frame color image I 1 The corresponding first coordinate system is transformed into the second frame color image I 2 The corresponding second coordinate system is calculated, and the average distance error between the pixel points of the first plane feature and the pixel points of the second plane feature in the second coordinate system is calculated. If the average distance error is less than the distance error threshold, it can be considered that the first plane feature PL 1 and the second plane feature PL 2 are the same plane features. The distance error threshold may include, for example, 0.01 m. The average error err ave The expression of is as follows, where p iThe first plane feature PL 1 A pixel on the i is the corresponding 3D space point, s is the depth value of the pixel found according to the depth image, K is the intrinsic parameter matrix of the camera, and · represents the dot multiplication operation of the vector:

[0111] P i =s*K -1 p i

[0112]

[0113] Step S305: matching the same plane features between the multiple frames of color images, and establishing the same-plane error constraints of the multiple frames of color images as the plane error constraints of the multiple frames of color images.

[0114] In some embodiments, when the same plane features between multiple frames of color images are obtained, the same plane features between the multiple frames of color images can be matched to establish the same-plane error constraints of the multiple frames of color images, and the established same-plane error constraints of the multiple frames of color images can be used as the plane error constraints of the multiple frames of color images.

[0115] Please continue reading Figure 5 , assuming that the two color image frames include the first color image I 1 and the second color image I 2 , the first frame color image I 1 The first plane feature PL 1 , its plane parameters are (a 1 ,b 1 ,c 1 ,d 1 ), where vector n 1 (a 1 ,b 1 ,c 1 ) is the normal vector of the first plane feature, d 1 is the distance from the origin to the first plane feature; the second frame color image I 2 The second plane feature PL 2 , its plane parameters are (a 2 ,b 2 ,c 2 ,d 2 ), where vector n 2 (a 2 ,b 2 ,c 2 ) is the normal vector of the second plane feature, d 2 is the distance from the origin to the second plane feature, the first frame color image I 1The rotation transformation to the world coordinate system is R w1 , the translation transformation is t w1 , the second frame color image I 2 The rotation transformation to the world coordinate system is R w2 , the translation transformation is t w2 .

[0116] The same plane error constraint is to transform the first frame color image I 1 The first plane feature PL under the corresponding first coordinate 1 The plane parameters are transformed into the second frame color image I after posture transformation. 2 The corresponding second coordinate and the second plane feature PL 2 The plane parameters are compared to form error constraints to optimize the rotation transformation R w1 , translation transformation t w1 , rotation transformation R w2 , translation transformation t w2 . The error expression is as follows:

[0117]

[0118] See also Figure 6 , Figure 6 This application shows Figure 4 The flowchart of step S305 of the method for determining the position and posture of the camera shown in FIG. Figure 6 The process shown is described in detail, and the method may specifically include the following steps:

[0119] Step S3051: Acquire parallel plane features between the multiple frames of color images, and acquire vertical plane features between the multiple frames of color images.

[0120] In this embodiment, when the plane features included in each of the multiple color image frames are obtained, the plane features included in each of the multiple color image frames can be matched to obtain parallel plane features between the multiple color image frames, and to obtain vertical plane features between the multiple color image frames. Among them, the reason for looking for the matching relationship between parallel plane features and vertical plane features is that in indoor environments, parallel and vertical planes appear frequently in scenes, such as corridors, corners, etc., and extracting these plane feature matching relationships is conducive to providing more sufficient constraints in the SLAM process. In some embodiments, when the plane features included in each of the multiple color image frames are obtained, the plane features included in each of the multiple color image frames can be matched to obtain parallel plane features from the plane features included in each of the multiple color image frames, and to obtain vertical plane features from the plane features included in each of the multiple color image frames.

[0121] In some embodiments, taking a multi-frame color image including a first frame color image and a second frame color image as an example, all plane features on the first frame color image can be traversed, the normal vector of each plane feature can be transferred to the coordinate system corresponding to the second frame color image, and the normal vector angles with the normal vectors of all plane features on the second frame color image are calculated respectively. If the angle is less than ±5, the two plane features can be considered to be parallel plane features.

[0122] In some embodiments, taking a multi-frame color image including a first frame color image and a second frame color image as an example, all plane features on the first frame color image can be traversed, the normal vector of each plane feature can be transferred to the coordinate system corresponding to the second frame color image, and the normal vector angles with the normal vectors of all plane features on the second frame color image are calculated respectively. If the angle is greater than 85° and less than 90°, the two plane features can be considered to be perpendicular plane features.

[0123] Step S3052: Matching parallel plane features between the multiple frames of color images to establish parallel plane error constraints for the multiple frames of color images.

[0124] In some implementations, when parallel plane features between multiple frames of color images are obtained, the parallel plane features between the multiple frames of color images may be matched to establish parallel plane error constraints of the multiple frames of color images.

[0125] Please continue reading Figure 5 , where the parallel plane features lack the constraint term of the distance from the origin compared to the same plane features. Therefore, only the constraint term between the normal vectors can be established to optimize the rotation relationship of the image frame. The specific process is as follows: Assume that the two color image frames include the first color image I 1 and the second color image I 2 , the first frame color image I 1 The first plane feature PL 1 , vector n 1 (a 1 ,b 1 ,c 1 ) is the normal vector of the first plane feature; the second frame color image I 2 The second plane feature PL 2 , its plane parameters are (a 2 ,b 2 ,c 2 ,d 2 ), vector n 2 (a 2 ,b 2 ,c 2 ) is the normal vector of the second plane feature, the first frame color image I 1The rotation transformation to the world coordinate system is R w1 , the second frame color image I 2 The rotation transformation to the world coordinate system is R w2 Based on this, the parallel plane error constraint of the parallel plane feature is to transform the first frame color image I 1 The first plane feature PL under the corresponding first coordinate 1 The normal vector is transformed into the second frame color image I after the posture transformation. 2 The corresponding second coordinate and the second plane feature PL 2 The normal vector is compared to form an error constraint to optimize the rotation transformation R w1 , rotation transformation R w2 . The error expression is as follows:

[0126] E 2 =1-n 2 ·(R w2 -1 *R w1 )*n 1

[0127] Step S3053: Match the vertical plane features between the multiple frames of color images to establish vertical plane error constraints for the multiple frames of color images.

[0128] In some implementations, when vertical plane features between multiple frames of color images are obtained, the vertical plane features between the multiple frames of color images may be matched to establish vertical plane error constraints of the multiple frames of color images.

[0129] Please continue reading Figure 5 , where the vertical plane feature lacks the constraint term of the distance from the origin compared to the same plane feature. Therefore, only the constraint term between the normal vectors can be established to optimize the rotation relationship of the image frame. The specific process is as follows: Assume that the two color image frames include the first color image I 1 and the second color image I 2 , the first frame color image I 1 The first plane feature PL 1 , vector n 1 (a 1 ,b 1 ,c 1 ) is the normal vector of the first plane feature; the second frame color image I 2 The second plane feature PL 2 , vector n 2 (a 2 ,b 2 ,c 2 ) is the normal vector of the second plane feature, the first frame color image I1 The rotation transformation to the world coordinate system is R w1 , the second frame color image I 2 The rotation transformation to the world coordinate system is R w2 Based on this, the vertical plane error constraint of the vertical plane feature is to transform the first frame color image I 1 The first plane feature PL under the corresponding first coordinate 1 The normal vector is transformed into the second frame color image I after the posture transformation. 2 The corresponding second coordinate and the second plane feature PL 2 The normal vector is compared to form an error constraint to optimize the rotation transformation R w1 , rotation transformation R w2 Due to the first plane feature PL 1 and the second plane feature PL 2 For a vertical relationship, the normal vector angle should ideally be 90°, cos90 = 0. The error expression is as follows:

[0130] E 3 =n 2 ·(R w2 -1 *R w1 )*n 1

[0131] Step S3054: Based on the same-plane error constraints of the multiple-frame color images, the parallel-plane error constraints of the multiple-frame color images, and the vertical-plane error constraints of the multiple-frame color images, the plane error constraints of the multiple-frame color images are obtained.

[0132] In some embodiments, when obtaining the same-plane error constraints of multiple-frame color images, the parallel plane error constraints of multiple-frame color images, and the vertical plane error constraints of multiple-frame color images, the plane error constraints of the multiple-frame color images can be obtained based on the same-plane error constraints of the multiple-frame color images, the parallel plane error constraints of the multiple-frame color images, and the vertical plane error constraints of the multiple-frame color images.

[0133] In some embodiments, after obtaining the co-planar error constraint Parallel plane error constraint Vertical plane error constraint Then the plane error constraints of the multi-frame color image can be obtained as follows:

[0134]

[0135] Step S306: Determine the position and posture of the camera based on the reprojection error constraint and the plane error constraint.

[0136] Among them, the reprojection error constraint can be Then based on the reprojection error constraint and the plane error constraint, the overall error constraint can be obtained as follows:

[0137]

[0138] Among them, after constructing the overall error constraint of the SLAM system, it can be optimized using nonlinear optimization methods to solve the optimal camera posture.

[0139] Compared with the method for determining the position and posture of a camera provided in an embodiment of the present application, Figure 1 The camera posture determination method shown in the embodiment also uses a two-stage method in the plane feature matching stage, first judging whether the normal vector angle of the feature to be matched meets the threshold requirement. If it does not meet the threshold requirement, there is no need to perform subsequent average distance verification, so that the calculation time of plane matching can be effectively saved through pre-screening. In addition, the embodiment not only designs matching schemes for the same plane under different viewing angles, but also designs matching schemes for mutually parallel and mutually perpendicular surfaces under different viewing angles, making full use of the geometric information in the operating environment.

[0140] See also Figure 7 , Figure 7 The module block diagram of the camera posture determination device provided in the embodiment of the present application is shown. Figure 7 As described in the block diagram shown in FIG. 2 , the camera posture determination device 200 includes: an image acquisition module 210, a reprojection error constraint establishment module 220, a plane error constraint establishment module 230, and a camera posture determination module 240, wherein:

[0141] The image acquisition module 210 is used to acquire multiple frames of color images and multiple frames of depth images obtained by photographing environmental reference objects with a camera.

[0142] The reprojection error constraint establishing module 220 is used to extract the feature points of the environmental reference object in the color image, and reproject the feature points of the multiple frames of color images to establish the reprojection error constraints of the multiple frames of color images.

[0143] The plane error constraint establishing module 230 is used to extract the plane features of the environmental reference object in the depth image as the plane features of the environmental reference object in the corresponding color image, and match the plane features of the multiple frames of color images to establish the plane error constraints of the multiple frames of color images.

[0144] Furthermore, the plane error constraint establishment module 230 includes: a depth image division submodule, an image fusion submodule, a first error acquisition submodule and a plane feature acquisition submodule, wherein:

[0145] The depth image division submodule is used to divide the depth image into multiple basic image blocks.

[0146] The image fusion submodule is used to select a basic image block from the multiple basic image blocks as a target image block, and fuse the target image block with the basic image blocks adjacent to it to obtain a fused image block.

[0147] Furthermore, the image fusion submodule includes: an image block filtering unit and a first image fusion unit, wherein:

[0148] The image block filtering unit is used to filter the basic image blocks that meet the first preset condition from the multiple basic image blocks to obtain multiple image blocks to be processed.

[0149] The first image fusion unit is used to select an image block to be processed from the multiple image blocks to be processed as a target image block, and fuse the target image block with the adjacent image blocks to be processed to obtain a fused image block.

[0150] Furthermore, the image fusion submodule includes: an angle acquisition unit and a second image fusion unit, wherein:

[0151] The angle acquisition unit is used to select a basic image block from the multiple basic image blocks as a target image block, and acquire the angle between the normal vector of the target image block and the normal vector of the basic image block adjacent to the target image block.

[0152] The second image fusion unit is used to fuse the target image block with the basic image block adjacent to the target image block to obtain a fused image block when the angle is smaller than the angle threshold.

[0153] A first error obtaining submodule is used to perform plane fitting on the fused image block and obtain an average error of the plane fitting as a first error;

[0154] The plane feature acquisition submodule is used to process the multiple basic image blocks based on the relationship between the first error and the error threshold until the multiple basic image blocks are processed, and then obtain the plane features of the environmental reference object in the depth image obtained by processing.

[0155] Furthermore, the plane feature acquisition submodule includes: a first plane feature acquisition unit and a second plane feature acquisition unit, wherein:

[0156] The first plane feature acquisition unit is used to, when the first error is less than an error threshold, use the fused image block as a target image block, and repeat the step of fusing the target image block with a basic image block adjacent to the target image block, until all the basic image blocks have been fused, and then determine the image block obtained by fusion as the plane feature of the environmental reference object in the depth image.

[0157] A second plane feature acquisition unit is used to reselect a basic image block from the multiple basic image blocks as a target image block when the first error is greater than or equal to an error threshold, and repeat the step of fusing the target image block with the basic image blocks adjacent to the target image block until all the multiple basic image blocks have been fused, and then determine the image block obtained by fusion as the plane feature of the environmental reference object in the depth image.

[0158] Furthermore, the plane feature acquisition submodule includes: an initial plane feature acquisition unit and a third plane feature acquisition unit, wherein:

[0159] The initial plane feature acquisition unit is used to acquire a plurality of initial plane features of the environmental reference object in the depth image obtained by processing.

[0160] The third plane feature acquisition unit is used to filter the initial plane features that meet the second preset condition from the multiple initial plane features to obtain the plane features of the environmental reference object in the depth image.

[0161] Furthermore, the plane error constraint establishing module 230 includes: a pixel point acquisition submodule, a second error acquisition submodule and a first plane error constraint establishing submodule, wherein:

[0162] The pixel point acquisition submodule is used to acquire pixel points located at the edge position of the plane feature.

[0163] The second error obtaining submodule is used to perform plane fitting on the pixel points and the plane features, and obtain the plane fitting error as the second error.

[0164] The first plane error constraint establishing submodule is used to determine that the pixel point belongs to the plane feature when the second error is less than the error threshold.

[0165] Furthermore, the plane error constraint establishing module 230 includes: a plane feature acquisition submodule and a second plane error constraint establishing submodule, wherein:

[0166] The plane feature acquisition submodule is used to acquire the same plane features between the multiple frames of color images.

[0167] The second plane error constraint establishing submodule is used to match the same plane features between the multiple frames of color images, and establish the same plane error constraints of the multiple frames of color images as the plane error constraints of the multiple frames of color images.

[0168] Furthermore, the second plane error constraint establishing submodule includes: a fourth plane feature acquisition unit, a third plane error constraint establishing unit, a fourth plane error constraint establishing unit and a fifth plane error constraint establishing unit, wherein:

[0169] The fourth plane feature acquisition unit is used to acquire parallel plane features between the multiple frames of color images and to acquire vertical plane features between the multiple frames of color images.

[0170] The third plane error constraint establishing unit matches the parallel plane features between the multiple frames of color images to establish the parallel plane error constraints of the multiple frames of color images.

[0171] The fourth plane error constraint establishing unit matches the vertical plane features between the multiple frames of color images to establish the vertical plane error constraints of the multiple frames of color images.

[0172] The fifth plane error constraint establishing unit is used to obtain the plane error constraints of the multiple frames of color images based on the same plane error constraints of the multiple frames of color images, the parallel plane error constraints of the multiple frames of color images, and the vertical plane error constraints of the multiple frames of color images.

[0173] Further, the multiple color image frames include a first color image frame and a second color image frame, the first color image frame includes a first plane feature, the second color image frame includes a second plane feature, and the plane error constraint establishment module 230 includes: a normal vector conversion submodule, a pixel point conversion submodule and a plane feature determination submodule, wherein:

[0174] A normal vector conversion submodule is used to convert the normal vector of the first plane feature from a first coordinate system to a second coordinate system based on the posture conversion relationship between the first frame color image and the second frame color image, wherein the first coordinate system corresponds to the first frame color image, and the second coordinate system corresponds to the second frame color image.

[0175] A pixel conversion submodule is used to convert the pixel points of the first plane feature from the first coordinate system to the second coordinate system based on the posture conversion relationship between the first frame color image and the second frame color image when the angle between the normal vector of the first plane feature and the normal vector of the second plane feature in the second coordinate system is less than an angle threshold.

[0176] The plane feature determination submodule is used to determine that the first plane feature and the second plane feature are the same plane feature when the average distance error between the pixel points of the first plane feature and the pixel points of the second plane feature in the second coordinate system is less than a distance error threshold.

[0177] The camera pose determination module 240 is used to determine the pose of the camera based on the reprojection error constraint and the plane error constraint.

[0178] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0179] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0180] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0181] See also Figure 8 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0182] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0183] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0184] See also Fig. 9 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0185] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 300 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0186] In summary, the camera posture determination method, device, electronic device and storage medium provided in the embodiments of the present application obtain multiple color images and multiple depth images obtained by shooting environmental reference objects with a camera, extract feature points of the environmental reference objects in the color images, and reproject the feature points of the multiple color images to establish a reprojection error of the multiple color images, extract the planar features of the environmental reference objects in the depth images as the corresponding planar features of the environmental reference objects in the color images, match the planar features of the multiple depth images, establish planar error constraints for the multiple color images, determine the camera posture based on the reprojection error constraints and the planar error constraints, and thus extract planar features through the depth images. In this way, the plane extraction will not be affected by the imaging quality of the color image, and can still stably propose effective planar features in dark, texture-free environments, and the positioning is more robust and accurate.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for determining the position and posture of a camera, characterized in that: The method comprises: Acquire multiple frames of color images and multiple frames of depth images obtained by photographing environmental reference objects with a camera; Extracting feature points of the environmental reference object in the color image, and reprojecting the feature points of the multiple frames of color images to establish reprojection error constraints of the multiple frames of color images; Extracting the plane features of the environmental reference object in the depth image as the plane features of the environmental reference object in the corresponding color image, and matching the plane features of the multiple frames of color images to establish plane error constraints of the multiple frames of color images; Determining a pose of the camera based on the reprojection error constraint and the plane error constraint; Wherein, extracting the planar features of the environmental reference object in the depth image includes: Dividing the depth image into a plurality of basic image blocks; Selecting a basic image block from the multiple basic image blocks as a target image block, and fusing the target image block with the basic image blocks adjacent to the target image block to obtain a fused image block; Performing plane fitting on the fused image block, and obtaining an average error of the plane fitting as a first error; Based on the relationship between the first error and the error threshold, the multiple basic image blocks are processed until the multiple basic image blocks are processed, and then the plane features of the environmental reference object in the depth image obtained by processing are obtained; The processing of the plurality of basic image blocks based on the relationship between the first error and the error threshold until the plurality of basic image blocks are processed is completed, and obtaining the plane features of the environmental reference object in the depth image obtained by the processing includes: When the first error is less than an error threshold, the fused image block is used as a target image block, and the step of fusing the target image block with a basic image block adjacent to the target image block is repeated until all the basic image blocks are fused, and the fused image block is determined as a planar feature of the environmental reference object in the depth image; or When the first error is greater than or equal to an error threshold, a basic image block is reselected from the multiple basic image blocks as a target image block, and the step of fusing the target image block with the basic image blocks adjacent to it is repeated until all the multiple basic image blocks have been fused, and the fused image block is determined as the planar feature of the environmental reference object in the depth image.

2. The method according to claim 1, characterized in that The step of selecting a basic image block from the multiple basic image blocks as a target image block, and fusing the target image block with the basic image blocks adjacent to the target image block to obtain a fused image block comprises: Filtering basic image blocks that meet a first preset condition from the multiple basic image blocks to obtain multiple image blocks to be processed; An image block to be processed is selected from the multiple image blocks to be processed as a target image block, and the target image block is fused with the adjacent image blocks to be processed to obtain a fused image block.

3. The method according to claim 2, characterized in that The first preset condition includes: There are pixels with invalid depth information; or There are adjacent pixels with sudden changes in depth information; or The average error of the plane fit is greater than the error threshold.

4. The method according to claim 1, characterized in that: The step of selecting a basic image block from the multiple basic image blocks as a target image block, and fusing the target image block with the basic image blocks adjacent to the target image block to obtain a fused image block comprises: Selecting a basic image block from the multiple basic image blocks as a target image block, and obtaining an angle between a normal vector of the target image block and a normal vector of a basic image block adjacent to the target image block; When the angle is smaller than the angle threshold, the target image block is fused with the basic image block adjacent to the target image block to obtain a fused image block.

5. The method according to claim 1, characterized in that After the plane features of the environment reference object in the depth image obtained by the acquisition process, the method further includes: Acquire pixel points located at edge positions of the plane features; Performing plane fitting on the pixel point and the plane feature, and obtaining a plane fitting error as a second error; When the second error is smaller than the error threshold, it is determined that the pixel point belongs to the plane feature.

6. The method according to claim 1, characterized in that The plane features of the environmental reference object in the depth image obtained by the acquisition process include: Acquire a plurality of initial planar features of the environmental reference object in the depth image obtained by processing; Initial plane features that meet a second preset condition are filtered from the multiple initial plane features to obtain plane features of the environmental reference object in the depth image.

7. The method according to claim 6, characterized in that The second preset condition includes: The number of pixels contained is less than the threshold; or The average distance of the included pixels to the corresponding initial plane features is greater than the distance threshold.

8. The method according to any one of claims 1 to 7, characterized in that: The matching of the plane features of the multiple frames of color images to establish the plane error constraints of the multiple frames of color images includes: Acquire the same plane features between the multiple frames of color images; The same plane features between the multiple frames of color images are matched, and the same plane error constraints of the multiple frames of color images are established as the plane error constraints of the multiple frames of color images.

9. The method according to claim 8, characterized in that The matching of the same plane features between the multiple color image frames to establish the same plane error constraints of the multiple color image frames as the plane error constraints of the multiple color image frames includes: Acquire parallel plane features between the multiple frames of color images, and acquire vertical plane features between the multiple frames of color images; Matching parallel plane features between the multiple frames of color images to establish parallel plane error constraints for the multiple frames of color images; Matching vertical plane features between the multiple frames of color images to establish vertical plane error constraints for the multiple frames of color images; Based on the same-plane error constraints of the multiple-frame color images, the parallel-plane error constraints of the multiple-frame color images, and the vertical-plane error constraints of the multiple-frame color images, the plane error constraints of the multiple-frame color images are obtained.

10. The method according to claim 8, characterized in that The multiple color image frames include a first color image frame and a second color image frame, the first color image frame includes a first plane feature, and the second color image frame includes a second plane feature. Before acquiring the same plane features between the multiple color image frames, the method further includes: Based on the posture conversion relationship between the first color image frame and the second color image frame, converting the normal vector of the first plane feature from a first coordinate system to a second coordinate system, wherein the first coordinate system corresponds to the first color image frame, and the second coordinate system corresponds to the second color image frame; When the angle between the normal vector of the first plane feature and the normal vector of the second plane feature in the second coordinate system is less than an angle threshold, based on a posture conversion relationship between the first frame color image and the second frame color image, converting the pixel points of the first plane feature from the first coordinate system to the second coordinate system; When an average distance error between pixel points of the first plane feature and pixel points of the second plane feature in the second coordinate system is less than a distance error threshold, it is determined that the first plane feature and the second plane feature are the same plane feature.

11. A camera posture determination device, characterized in that: The device comprises: An image acquisition module is used to acquire multiple color images and multiple depth images obtained by shooting environmental reference objects with a camera; A reprojection error constraint establishment module, used for extracting feature points of the environmental reference object in the color image, and reprojecting the feature points of the multiple frames of color images to establish reprojection error constraints of the multiple frames of color images; A plane error constraint establishment module, used to extract the plane features of the environmental reference object in the depth image as the plane features of the environmental reference object in the corresponding color image, and match the plane features of the multiple frames of color images to establish the plane error constraints of the multiple frames of color images; A camera pose determination module, used to determine the pose of the camera based on the reprojection error constraint and the plane error constraint; Wherein, the plane error constraint establishment module includes: A depth image division submodule, used to divide the depth image into a plurality of basic image blocks; An image fusion submodule, configured to select a basic image block from the multiple basic image blocks as a target image block, and fuse the target image block with the basic image blocks adjacent to the target image block to obtain a fused image block; A first error obtaining submodule is used to perform plane fitting on the fused image block and obtain an average error of the plane fitting as a first error; a plane feature acquisition submodule, configured to process the plurality of basic image blocks based on a relationship between the first error and an error threshold, and to acquire plane features of the environmental reference object in the depth image obtained by processing when the plurality of basic image blocks are all processed; Wherein, the plane feature acquisition submodule includes: a first plane feature acquisition unit, configured to, when the first error is less than an error threshold, use the fused image block as a target image block, and repeat the step of fusing the target image block with a basic image block adjacent to the target image block, until all the basic image blocks have been fused, and then determine the image block obtained by fusion as a plane feature of the environmental reference object in the depth image; or A second plane feature acquisition unit is used to reselect a basic image block from the multiple basic image blocks as a target image block when the first error is greater than or equal to an error threshold, and repeat the step of fusing the target image block with the basic image blocks adjacent to the target image block until all the multiple basic image blocks have been fused, and then determine the image block obtained by fusion as the plane feature of the environmental reference object in the depth image.

12. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 10.

Images