Simultaneous localization and mapping backend optimization method and device, and storage medium

Abstract

The application provides a simultaneous localization and mapping backend optimization method and device and a storage medium. The method determines a target three-dimensional space point in three-dimensional space points of multiple key frames positioned by a simultaneous localization and mapping system, and then determines a re-projection error according to the target three-dimensional space point. A weight is provided by a maximum parallax corresponding to the target three-dimensional space point after preprocessing, and a relative re-projection error is obtained. The relative re-projection error describes the difference between the spatial position of a point and the spatial position of a camera, thereby providing a more robust residual error construction for the optimization problem of simultaneous localization and mapping, better performing spatial positioning, and solving the problem that the existing backend optimization cannot well represent the relationship between the re-projection error and the depth of a point, lacks the description of spatial information, and results in a low accuracy of the optimized pose estimation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and storage medium for simultaneous localization and mapping backend optimization. Background Technology

[0002] With the development of computer vision technology, simultaneous localization and mapping (SLAM) algorithms have been widely used in fields such as augmented reality, virtual reality, autonomous driving, and positioning and navigation of robots or drones.

[0003] In simultaneous localization and mapping (SLAM) algorithms, the problem can be divided into two parts: the front-end and the back-end. The main function of the front-end is to calculate the relative relationships between frames, such as the relative pose between frames. The main function of the back-end is to optimize the output of the front-end to obtain the optimal pose estimate. In visual SLAM methods, back-end optimization typically uses the reprojection error between keypoints matching between frames as a loss function, and then performs an overall optimization based on this loss function. For example, when the loss function is small, an optimized pose estimate is obtained.

[0004] However, the aforementioned backend optimizations cannot adequately represent the relationship between reprojection error and point depth, lacking spatial information description, which may lead to lower accuracy in the optimized pose estimation. Summary of the Invention

[0005] To address the problems existing in the prior art, this application provides a method, apparatus, and storage medium for optimizing the backend of synchronous positioning and mapping.

[0006] In a first aspect, embodiments of this application provide a method for optimizing the backend of simultaneous localization and mapping, including:

[0007] Determine the target three-dimensional spatial point from the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system;

[0008] Based on the target three-dimensional spatial points, determine the reprojection error;

[0009] The relative reprojection error is determined based on the reprojection error and the maximum disparity corresponding to the target three-dimensional space point after preprocessing. The preprocessing includes the operation of removing the rotation effect.

[0010] The synchronous positioning and mapping system performs backend optimization based on the relative reprojection error.

[0011] In one possible implementation, before determining the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the preprocessed target 3D spatial point, the method further includes:

[0012] Determine multiple target keyframes corresponding to the target's three-dimensional spatial points;

[0013] Based on the key point coordinates of key frame ui, key point coordinates of key frame uj, intrinsic parameter matrix of image acquisition device in the synchronous positioning and mapping system, and rotation matrix from key frame uj to key frame ui, multiple disparities corresponding to the target 3D space point after preprocessing are determined, wherein key frame ui and key frame uj are any two key frames among the multiple target key frames, i = 1, 2, ... m, j = 1, 2, ... m, and m is equal to the number of the multiple target key frames;

[0014] The maximum disparity is determined from the multiple disparities corresponding to the target three-dimensional space point after the preprocessing.

[0015] In one possible implementation, determining the multiple disparities corresponding to the preprocessed target 3D spatial points based on the keypoint coordinates of keyframe ui, the keypoint coordinates of keyframe uj, the intrinsic parameter matrix of the image acquisition device in the synchronous positioning and mapping system, and the rotation matrix from keyframe uj to keyframe ui includes:

[0016] Calculate the product of the intrinsic parameter matrix, the rotation matrix, the inverse of the intrinsic parameter matrix, and the keypoint coordinates of the keyframe uj;

[0017] Based on the difference between the keypoint coordinates of the keyframe UI and the product, multiple disparities corresponding to the target 3D spatial points after preprocessing are determined.

[0018] In one possible implementation, determining the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the preprocessed target 3D spatial point includes:

[0019] Calculate the ratio of the reprojection error to the maximum parallax;

[0020] The ratio is taken as the relative reprojection error.

[0021] In one possible implementation, determining the reprojection error based on the target three-dimensional space point includes:

[0022] Determine the position obtained by projecting the target three-dimensional spatial point;

[0023] The reprojection error is determined based on the position obtained from the projection.

[0024] In one possible implementation, the backend optimization of the synchronous positioning and mapping system based on the relative reprojection error includes:

[0025] Determine whether the relative reprojection error reaches a preset error threshold;

[0026] If the relative reprojection error does not reach the preset error threshold, the step of determining the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system is re-executed so that the relative reprojection error reaches the preset error threshold. Based on the relative reprojection error, the backend optimization of the synchronous positioning and mapping system is performed.

[0027] In one possible implementation, before determining the target 3D spatial point among the 3D spatial points of multiple keyframes located in the synchronous positioning and mapping system, the method further includes:

[0028] The synchronous positioning and mapping system is initialized.

[0029] In one possible implementation, initializing the synchronous positioning and mapping system includes:

[0030] Acquire a preset number of consecutive frame images, and perform the preprocessing on the preset number of consecutive frame images;

[0031] Using a pre-built, adaptively sized sliding window, initial keyframes are selected from a preset number of consecutive frame images after removing the effects of rotation. The initial keyframes include multiple keyframes.

[0032] Based on the aforementioned keyframes, synchronous positioning and mapping initialization are performed.

[0033] In one possible implementation, the synchronous localization and mapping initialization based on the multiple keyframes includes:

[0034] Determine the relative pose of the first keyframe and the last keyframe among the plurality of keyframes;

[0035] Based on the relative pose of the first keyframe and the last keyframe, obtain the three-dimensional spatial points of each keyframe in the plurality of keyframes.

[0036] The relative pose of each key frame in the plurality of key frames is determined based on the relative pose of the first key frame and the last key frame, and the three-dimensional spatial points of each key frame in the plurality of key frames.

[0037] The initial map is established based on the 3D spatial points of each keyframe in the optimized initial keyframe and the relative poses of each keyframe in the initial keyframe.

[0038] Secondly, embodiments of this application provide a synchronous positioning and mapping backend optimization device, comprising:

[0039] The spatial point determination module is used to determine the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system.

[0040] The reprojection error determination module is used to determine the reprojection error based on the target three-dimensional spatial points;

[0041] The relative reprojection error determination module is used to determine the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the target three-dimensional space point after preprocessing. The preprocessing includes the operation of removing the rotation effect.

[0042] The synchronous positioning and mapping system undergoes backend optimization. The backend optimization module is used to optimize the system based on the relative reprojection error.

[0043] In one possible implementation, an error determination module is also included, which is used to determine multiple target keyframes corresponding to the target three-dimensional space point before the relative reprojection error determination module determines the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the preprocessed target three-dimensional space point.

[0044] Based on the key point coordinates of key frame ui, key point coordinates of key frame uj, intrinsic parameter matrix of image acquisition device in the synchronous positioning and mapping system, and rotation matrix from key frame uj to key frame ui, multiple disparities corresponding to the target 3D space point after preprocessing are determined, wherein key frame ui and key frame uj are any two key frames among the multiple target key frames, i = 1, 2, ... m, j = 1, 2, ... m, and m is equal to the number of the multiple target key frames;

[0045] The maximum disparity is determined from the multiple disparities corresponding to the target three-dimensional space point after the preprocessing.

[0046] In one possible implementation, the error determination module is specifically used for:

[0047] Calculate the product of the intrinsic parameter matrix, the rotation matrix, the inverse of the intrinsic parameter matrix, and the keypoint coordinates of the keyframe uj;

[0048] Based on the difference between the keypoint coordinates of the keyframe UI and the product, multiple disparities corresponding to the target 3D spatial points after preprocessing are determined.

[0049] In one possible implementation, the relative reprojection error determination module is specifically used for:

[0050] Calculate the ratio of the reprojection error to the maximum parallax;

[0051] The ratio is taken as the relative reprojection error.

[0052] In one possible implementation, the reprojection error determination module is specifically used for:

[0053] Determine the position obtained by projecting the target three-dimensional spatial point;

[0054] The reprojection error is determined based on the position obtained from the projection.

[0055] In one possible implementation, the backend optimization module is specifically used for:

[0056] Determine whether the relative reprojection error reaches a preset error threshold;

[0057] If the relative reprojection error does not reach the preset error threshold, the spatial point determination module re-executes the step of determining the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located in the synchronous positioning and mapping system, so that the relative reprojection error reaches the preset error threshold. The back-end optimization module performs back-end optimization on the synchronous positioning and mapping system based on the relative reprojection error.

[0058] In one possible implementation, a synchronous positioning and mapping initialization module is also included, which is used to initialize the synchronous positioning and mapping system before the spatial point determination module determines the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system.

[0059] In one possible implementation, the synchronous positioning and mapping initialization module is specifically used for:

[0060] Acquire a preset number of consecutive frame images, and perform the preprocessing on the preset number of consecutive frame images;

[0061] Using a pre-built, adaptively sized sliding window, initial keyframes are selected from a preset number of consecutive frame images after removing the effects of rotation. The initial keyframes include multiple keyframes.

[0062] Based on the aforementioned keyframes, synchronous positioning and mapping initialization are performed.

[0063] In one possible implementation, the synchronous positioning and mapping initialization module is specifically used for:

[0064] Determine the relative pose of the first keyframe and the last keyframe among the plurality of keyframes;

[0065] Based on the relative pose of the first keyframe and the last keyframe, obtain the three-dimensional spatial points of each keyframe in the plurality of keyframes.

[0066] The relative pose of each key frame in the plurality of key frames is determined based on the relative pose of the first key frame and the last key frame, and the three-dimensional spatial points of each key frame in the plurality of key frames.

[0067] The initial map is established based on the 3D spatial points of each keyframe in the optimized initial keyframe and the relative poses of each keyframe in the initial keyframe.

[0068] Thirdly, embodiments of this application provide a synchronous positioning and mapping backend optimization device, comprising:

[0069] processor;

[0070] Memory; and

[0071] Computer programs;

[0072] The computer program is stored in the memory and configured to be executed by the processor, the computer program including instructions for performing the method as described in the first aspect.

[0073] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that causes a server to perform the method described in the first aspect.

[0074] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which are executed by a processor according to the method described in the first aspect.

[0075] The synchronous positioning and mapping backend optimization method, apparatus, and storage medium provided in this application embodiment determine a target 3D spatial point from among the 3D spatial points of multiple keyframes located by the synchronous positioning and mapping system. Then, based on the target 3D spatial point, a reprojection error is determined, and a relative reprojection error is determined based on the reprojection error and the maximum disparity corresponding to the preprocessed target 3D spatial point. The preprocessing includes removing rotation effects. Based on the relative reprojection error, the synchronous positioning and mapping system is optimized at the backend. The disparity is the baseline length between the projection points of the 3D spatial point on the camera frame. This application embodiment uses the disparity to provide weights for the reprojection error to obtain the relative reprojection error. This relative reprojection error describes the difference between the spatial position of the point and the spatial position of the camera, thus providing a more robust residual construction for the synchronous positioning and mapping optimization problem, better enabling spatial positioning. This solves the problem that existing backend optimizations cannot well represent the relationship between the reprojection error and the depth of the point, lack spatial information description, and result in low accuracy of the optimized pose estimation. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 A schematic diagram of a synchronous positioning and mapping backend optimization system architecture provided in this application embodiment;

[0078] Figure 2 A flowchart illustrating a synchronous positioning and mapping backend optimization method provided in an embodiment of this application;

[0079] Figure 3 A schematic diagram of reprojection error provided in an embodiment of this application;

[0080] Figure 4 A flowchart illustrating another synchronous positioning and mapping backend optimization method provided in an embodiment of this application;

[0081] Figure 5 A flowchart illustrating another synchronous positioning and mapping backend optimization method provided in an embodiment of this application;

[0082] Figure 6 This is a schematic diagram illustrating the optimization of the synchronous positioning and mapping backend provided in an embodiment of this application.

[0083] Figure 7 This is a schematic diagram of the structure of a synchronous positioning and mapping backend optimization device provided in an embodiment of this application;

[0084] Figure 8 A schematic diagram of another synchronous positioning and mapping backend optimization device provided in an embodiment of this application;

[0085] Figure 9 A schematic diagram of another synchronous positioning and mapping backend optimization device provided in an embodiment of this application;

[0086] Figure 10A A schematic diagram of the basic hardware architecture of a synchronous positioning and mapping backend optimization device provided in this application;

[0087] Figure 10B A schematic diagram of the basic hardware architecture of another synchronous positioning and mapping backend optimization device provided in this application;

[0088] Figure 10C A schematic diagram of the basic hardware architecture of another synchronous positioning and mapping backend optimization device provided in this application. Detailed Implementation

[0089] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0090] The terms “first,” “second,” “third,” and “fourth,” etc. (if present), in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0091] In related technologies, taking the simultaneous localization and mapping (SLR) algorithm in mobile devices as an example, the SLR algorithm is used to obtain the mobile device's own pose, the environment in which the mobile device is located, and the mobile device's position in the environment. When a user uses the mobile device, the front end of the SLR algorithm calculates the relative relationships between frames, such as calculating the relative pose between frames. The back end's main role is to optimize the output of the front end to obtain the optimal pose estimate. Specifically, in the back end optimization of the SLR algorithm, the reprojection error between keypoints matching between frames is typically used as the loss function. An overall optimization is then performed based on this loss function; for example, a smaller loss function yields an optimized pose estimate. The accuracy of this pose estimate affects the effectiveness of mobile devices in implementing augmented reality, virtual reality, and autonomous driving applications based on the SLR algorithm.

[0092] The reprojection error is the error obtained by comparing the pixel coordinates (the observed projected position) with the position obtained by projecting the 3D point according to the currently estimated pose. For example, with the same camera motion, points farther away have smaller motion distances on the image, while points closer have larger motion distances. If the reprojection error is used as the loss function for simultaneous localization and mapping (SLR) backend optimization, when the loss function is small, the SLR backend optimization cannot accurately represent the relationship between the reprojection error and the point's depth, lacking spatial information description. Therefore, it cannot be determined whether the optimized result is good, which may lead to a lower accuracy in the optimized pose estimation.

[0093] Therefore, this application proposes a synchronous localization and mapping backend optimization method. By using disparity to provide weights for reprojection errors, a relative reprojection error is obtained. Here, disparity is the baseline length between the projection points of a 3D spatial point on the camera frame. Thus, the aforementioned relative reprojection error can describe the difference between the spatial position of the point and the spatial position of the camera, providing a more robust residual construction for the optimization problem of synchronous localization and mapping. This allows for better spatial localization and solves the problem that existing backend optimizations cannot well represent the relationship between reprojection error and point depth, lack spatial information description, and may lead to low accuracy of the optimized pose estimation.

[0094] Optionally, the synchronous positioning and mapping backend optimization method provided in this application embodiment can be applied to, for example, Figure 1 In the application scenarios shown. Figure 1 This application merely illustrates one possible application scenario of the synchronous positioning and mapping backend optimization method provided in this embodiment. The application scenarios of the synchronous positioning and mapping backend optimization method provided in this embodiment are not limited to... Figure 1 The application scenarios shown.

[0095] Figure 1 A schematic diagram illustrating the optimized system architecture for synchronous positioning and mapping backend. Figure 1 Taking a user processing video on a mobile device as an example, where the mobile device can be a phone or tablet, etc., the architecture may include an acquisition unit 101, a processor 102, and a display unit 103.

[0096] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the backend optimization architecture of synchronous positioning and mapping. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1The components shown can be implemented in hardware, software, or a combination of both.

[0097] Taking a mobile phone as an example, the acquisition unit 101 can be a camera on the phone. The user can capture video using the phone's camera and then send the captured video to the processor 102 for processing. Here, the acquisition unit 101 can be a camera, an input / output interface, or a communication interface. The user can receive video and other information sent by other users through this interface and send the received video to the processor 102 for processing.

[0098] In the specific implementation process, after acquiring the aforementioned video, the processor 102 can use a simultaneous localization and mapping (SLT / Map) system to locate the 3D spatial points of multiple keyframes in the video. Taking a target 3D spatial point as an example, this target 3D spatial point can be any one of the aforementioned 3D spatial points. The processor 102 can determine the key value that can be observed from the 3D spatial points of multiple keyframes located by the SLT / Map system. This observed value is the projection of the target 3D spatial point. Furthermore, based on the determination of the target 3D spatial point, the reprojection error is determined, and the maximum disparity corresponding to the target 3D spatial point after preprocessing is used to provide weights for the reprojection error to obtain the relative reprojection error. This preprocessing includes operations to remove the effects of rotation. Here, the disparity is the baseline length between the projection points of the 3D spatial point on the camera frame. Therefore, the relative reprojection error describes the difference between the spatial position of the point and the spatial position of the camera, thereby providing a more robust residual construction for the optimization problem of SLT / Map, enabling better spatial positioning and improving the accuracy of the optimized pose estimation.

[0099] The display unit 103 can be used to display the aforementioned target three-dimensional spatial points, reprojection errors, parallax, and relative reprojection errors. The display unit can also be a touch screen, used to receive user commands while displaying the aforementioned content, thereby enabling interaction with the user.

[0100] It should be understood that the system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0101] The technical solutions of this application are described below using several embodiments as examples. The same or similar concepts or processes may not be repeated in some embodiments.

[0102] Figure 2This is a flowchart illustrating a synchronous positioning and mapping backend optimization method provided in an embodiment of this application. The execution entity of this embodiment can be... Figure 1 The processor 102 in the application can be specifically executed based on the actual application scenario; this embodiment does not impose any particular restrictions on this. Figure 2 As shown, the synchronous positioning and mapping backend optimization method provided in this application embodiment may include the following steps:

[0103] S201: Determine the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system.

[0104] Here, the processor can first determine any one of the three-dimensional spatial points in the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system as the target three-dimensional spatial point. Then, it can also determine the key value that can be observed of the target three-dimensional spatial point. This observation value is the projection of the target three-dimensional spatial point. In addition, it can determine the multiple target key frames corresponding to the target three-dimensional spatial point.

[0105] S202: Determine the reprojection error based on the above target three-dimensional spatial points.

[0106] For example, the processor described above can use the perspective N-point method to determine the position obtained by projecting the target three-dimensional space point, and then determine the reprojection error based on the position obtained by the projection.

[0107] The perspective N-point method is used to solve the problem of camera pose estimation when the coordinates of three-dimensional points in a known partial world coordinate system and their two-dimensional camera coordinate system are known. In the embodiments of this application, the processor uses the perspective N-point method to determine the position obtained by projecting the target three-dimensional point, and then determines the reprojection error based on the position.

[0108] Here, the reprojection error is the error obtained by comparing the pixel coordinates (the observed projected position) with the position obtained by projecting the 3D point according to the currently estimated pose (e.g., the position obtained by projecting the target 3D spatial point mentioned above).

[0109] The aforementioned reprojection error is the reprojection error after removing the effects of rotation.

[0110] The above reprojection error calculation is as follows: Figure 3 As shown, the observations p1 and p2 are projections of the same spatial point p. There is a certain distance between the observed value p2 and the observed value p2, which is the reprojection error.

[0111] Considering n points P in three-dimensional space and their projections p, the calculation of R and T can be expressed as ξ. Assume a certain point p in space... i=[X i ,Y i Z i ] T Its projected pixel coordinates are u i =[u i ,v i ] T

[0112] The relationship between pixel position and spatial point position is as follows:

[0113]

[0114] Among them, s i Let R be the distance (depth), k be the camera intrinsic parameter matrix, R be the rotation matrix, and T be the translation matrix.

[0115] Correspondingly, in matrix form, it is: s i u i =kexp(ξ^)p i

[0116] Due to the unknown camera pose and noise at the observation point, this equation contains an error. We can sum this error, construct a least-squares problem, and then find the optimal camera pose to minimize it.

[0117]

[0118] This can be solved using the Gauss-Newton method / Levonberg-Marquardt method.

[0119] S203: Based on the above reprojection error and the maximum parallax corresponding to the above target three-dimensional space point after preprocessing, determine the relative reprojection error. The preprocessing includes the operation of removing the rotation effect.

[0120] The reason for the preprocessing mentioned above is that rotation affects the pixel distance difference between frames, which in turn affects the disparity between frames, making it impossible for the synchronous localization and mapping backend optimization to obtain the optimal pose estimation. Therefore, in order to solve this problem, this embodiment of the application performs the above preprocessing, uses the reprojection error and the maximum disparity corresponding to the preprocessed target 3D space point to determine the relative reprojection error, performs synchronous localization and mapping backend optimization, and obtains the optimal pose estimation.

[0121] The processor can obtain the rotation information from the inertial measurement unit, and then determine the pixel distance difference of the rotation-affected frame based on the obtained information, determine the maximum disparity corresponding to the target three-dimensional space point after removing the rotation effect, further determine the relative reprojection error, and perform back-end optimization of the synchronous positioning and mapping system.

[0122] For example, the processor described above can calculate the ratio of the reprojection error to the maximum disparity and use this ratio as the relative reprojection error. Here, disparity is the baseline length between the projected points of a 3D spatial point on the camera frame. This allows the relative reprojection error to describe the difference between the spatial position of the point and the spatial position of the camera, providing a more robust residual construction for the optimization of simultaneous localization and mapping, and enabling better spatial localization.

[0123] S204: Based on the aforementioned relative reprojection error, perform backend optimization on the aforementioned synchronous positioning and mapping system.

[0124] In this embodiment, after determining the relative reprojection error, the processor can construct an optimization problem using the reprojection error as a loss function. Based on this optimization problem, the synchronous positioning and mapping system is then optimized at the backend to obtain an optimized pose estimate. For example, the processor can determine whether the relative reprojection error reaches a preset error threshold (this preset error threshold can be determined based on actual conditions, for example, equal to the minimum relative reprojection error). If the relative reprojection error does not reach the preset error threshold, the processor can re-execute the step of determining the target 3D spatial point among the 3D spatial points of multiple keyframes located by the synchronous positioning and mapping system, so that the relative reprojection error reaches the preset error threshold. Then, based on the relative reprojection error, the synchronous positioning and mapping system is optimized at the backend.

[0125] Furthermore, when performing backend optimization on the aforementioned simultaneous localization and mapping (SLTMA) system, the processor, in addition to considering the relative reprojection error, can construct an optimization problem based on the relative reprojection error, and then perform backend optimization on the SLTMA system based on this optimization problem. Alternatively, the processor can also consider the reprojection error, that is, it can simultaneously construct an optimization problem based on both the reprojection error and the relative error, and then perform backend optimization on the SLTMA system based on this optimization problem to obtain the optimal pose estimation and achieve more robust map construction.

[0126] In this embodiment, the processor determines a target 3D spatial point from the 3D spatial points of multiple keyframes located by the synchronous localization and mapping system. Then, based on the target 3D spatial point, it determines the reprojection error, and based on the reprojection error and the maximum disparity corresponding to the preprocessed target 3D spatial point, it determines the relative reprojection error. The preprocessing includes removing rotation effects. Based on the relative reprojection error, the synchronous localization and mapping system is optimized at the backend. Here, the disparity is the baseline length between the projection points of the 3D spatial point on the camera frame. This embodiment uses the disparity to provide weights for the reprojection error to obtain the relative reprojection error. This relative reprojection error describes the difference between the spatial position of the point and the spatial position of the camera, thus providing a more robust residual construction for the optimization problem of synchronous localization and mapping, better enabling spatial localization. This solves the problem that existing backend optimizations cannot well represent the relationship between the reprojection error and the depth of the point, lacking spatial information description, which may lead to low accuracy in the optimized pose estimation.

[0127] In addition, before determining the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the preprocessed target 3D spatial point, this embodiment of the application also considers determining multiple disparities corresponding to the preprocessed target 3D spatial point. Thus, the maximum disparity is determined from the multiple disparities. Based on the reprojection error and the maximum disparity, the relative reprojection error is determined. Based on the relative reprojection error, the backend optimization of the synchronous positioning and mapping system is performed to obtain the optimal pose estimation. Figure 4 This is a flowchart illustrating another synchronous positioning and mapping backend optimization method proposed in an embodiment of this application. Figure 4 As shown, the method includes:

[0128] S401: Determine the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system.

[0129] S402: Determine the reprojection error based on the above target three-dimensional spatial points.

[0130] The implementation of steps S401-S402 is the same as that of steps S201-S202 above, and will not be repeated here.

[0131] S403: Determine multiple target keyframes corresponding to the above target 3D spatial points.

[0132] S404: Based on the key point coordinates of key frame ui, key point coordinates of key frame uj, the intrinsic parameter matrix of the image acquisition device in the above synchronous positioning and mapping system, and the rotation matrix from key frame uj to key frame ui, determine multiple disparities corresponding to the target three-dimensional space points after preprocessing. The preprocessing includes the operation of removing rotation effects. Key frame ui and key frame uj are any two key frames among the above multiple target key frames, i = 1, 2, ... m, j = 1, 2, ... m, and m is equal to the number of the above multiple target key frames.

[0133] S405: Determine the maximum disparity from the multiple disparities corresponding to the target three-dimensional space point after the above preprocessing.

[0134] In addition, the processor can first calculate the product of the intrinsic parameter matrix, rotation matrix, inverse matrix of intrinsic parameter matrix, and key point coordinates of key frame uj. Then, based on the difference between the key point coordinates of key frame ui and the product, it can determine multiple disparities corresponding to the target three-dimensional space point after the preprocessing. From these multiple disparities, it can determine the maximum disparity.

[0135] For example, taking keyframe ui as the first frame and keyframe uj as the second frame, the processor calculates the parallax corresponding to the preprocessed target 3D space point as follows:

[0136] parallax=u1-k*R*k inv *u2

[0137] Where u1 represents the coordinates of key points in the first frame of the aforementioned multiple target keyframes; u2 represents the coordinates of key points in the second frame of the aforementioned multiple target keyframes; k is the intrinsic parameter matrix of the image acquisition device in the synchronous positioning and mapping system, such as the intrinsic parameter matrix of the camera; k inv R is the inverse of the intrinsic parameter matrix; R is the rotation matrix from the second frame to the first frame.

[0138] For example, the processor described above can calculate the ratio of the reprojection error to the maximum parallax mentioned above, and then use this ratio as the relative reprojection error.

[0139] S406: Determine the relative reprojection error based on the above-mentioned reprojection error and the above-mentioned maximum parallax.

[0140] S407: Based on the aforementioned relative reprojection error, perform backend optimization on the aforementioned synchronous positioning and mapping system.

[0141] The implementation of steps S406-S407 is the same as that of steps S203-S204 above, and will not be repeated here.

[0142] In this embodiment, before determining the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the preprocessed target 3D spatial point, the processor further considers determining multiple disparities corresponding to the preprocessed target 3D spatial point. From these multiple disparities, the maximum disparity is determined. Based on the reprojection error and the maximum disparity, the relative reprojection error is determined. Then, based on this relative reprojection error, the backend optimization of the synchronous localization and mapping system is performed to obtain the optimal pose estimation. Furthermore, the processor uses the disparity to provide weights for the reprojection error, obtaining the relative reprojection error. This relative reprojection error describes the difference between the spatial position of the point and the spatial position of the camera, thus providing a more robust residual construction for the optimization problem of synchronous localization and mapping, enabling better spatial localization. This solves the problem that existing backend optimizations cannot well represent the relationship between the reprojection error and the depth of the point, lacking spatial information description, resulting in low accuracy of the optimized pose estimation.

[0143] In addition, in this embodiment of the application, before determining the target three-dimensional spatial point from the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system, the synchronous positioning and mapping system is initialized to ensure that subsequent processing proceeds normally. Figure 5 This is a flowchart illustrating another synchronous positioning and mapping backend optimization method proposed in an embodiment of this application. Figure 5 As shown, the method includes:

[0144] S501: Initialize the synchronous positioning and mapping system.

[0145] Here, the processor can first acquire a preset number of consecutive frame images, perform the aforementioned preprocessing on the preset number of consecutive frame images, and then, using a pre-constructed adaptive-size sliding window, select initial keyframes from the preset number of consecutive frame images after removing the effects of rotation. The initial keyframes include multiple keyframes, and thus, based on the multiple keyframes, perform synchronous positioning and mapping initialization.

[0146] For example, the processor can determine the relative pose of the first keyframe and the last keyframe in the initial keyframe, and obtain the three-dimensional spatial points of each keyframe in the plurality of keyframes based on the relative pose of the first keyframe and the last keyframe. Based on the relative pose of the first keyframe and the last keyframe, and the three-dimensional spatial points of each keyframe in the plurality of keyframes, the processor can determine the relative pose of each keyframe in the plurality of keyframes. Finally, based on the three-dimensional spatial points of each keyframe in the plurality of keyframes and the relative pose of each keyframe in the plurality of keyframes, the processor can establish an initial map and complete the initialization of the synchronous localization and mapping system.

[0147] The preset number of consecutive frames can be determined based on actual conditions, such as frames 1, 2, ..., 25 in a video. The processor can pre-construct an adaptive-size sliding window, the size of which is adjustable and can be determined based on actual conditions, such as the size of 5 frames minus the size of 10 frames. Using this sliding window, the processor selects initial keyframes from the preset number of consecutive frames after removing rotation effects. For example, if the current sliding window length is 5 frames, the processor uses this window to select the preset number of consecutive frames after removing rotation effects, such as frames 1, 2, ..., 25 in the video, selecting frames 6, 7, 10, 12, and 13 as the initial keyframes.

[0148] The reason for the above preprocessing is that rotation affects the pixel distance difference between frames, but rotation alone is insufficient for synchronous positioning and mapping initialization. Therefore, in order to solve this problem, this embodiment of the application performs the above preprocessing and uses the pixel distance difference after removing the influence of rotation to filter the initial keyframes within the window, ensuring that there is sufficient pixel distance difference between frames in the window for synchronous positioning and mapping initialization, provided that there is sufficient common vision between the frames.

[0149] The aforementioned processor performs synchronous localization and mapping initialization based on initial keyframes selected from a certain number of consecutive frame images, reducing the time required for synchronous localization and mapping initialization. Furthermore, by utilizing pixel distance differences that remove the effects of rotation to select initial keyframes within a window, the processor ensures sufficient pixel distance differences between frames within the window for synchronous localization and mapping initialization, while simultaneously reducing the impact of rotation on synchronous localization and mapping initialization, thus improving the accuracy of synchronous localization and mapping initialization.

[0150] Furthermore, due to the known rotation and the lack of objective scale, the processor can perform synchronous positioning and mapping initialization using only two frames after selecting the initial keyframes. In this embodiment, to ensure sufficient pixel distance difference between frames within the window while maintaining sufficient common view, the processor can utilize the first and last keyframes from the initial keyframes for synchronous positioning and mapping initialization.

[0151] For example, the processor can first extract two-dimensional key points from the first key frame and the last key frame to obtain a two-dimensional key point of the first key frame and a two-dimensional key point of the last key frame. Then, using the two-dimensional key point of the first key frame and the two-dimensional key point of the last key frame, the relative pose of the first key frame and the last key frame can be determined.

[0152] Furthermore, the processor can use a two-dimensional key point of the first key frame and a two-dimensional key point of the last key frame to determine the essential matrix corresponding to the first key frame and the last key frame. Then, based on the essential matrix, it can obtain the rotation matrix R and the translation matrix T. Thus, based on the rotation matrix R and the translation matrix T, it can determine the relative pose of the first key frame and the last key frame.

[0153] For example, the processor can first determine the essential matrices corresponding to the first and last keyframes, and then obtain the rotation matrix R and translation matrix T based on these essential matrices. Thus, the relative pose of the first and last keyframes can be determined based on the rotation matrix R and translation matrix T. Specifically, the processor can use a random consistency sampling method to determine the essential matrices corresponding to the first and last keyframes, and then solve for the rotation matrix R and translation matrix T from these essential matrices using singular value decomposition. Here, the rotation matrix R and translation matrix T are the camera's pose parameters, and the rotation matrix R is known. Therefore, the processor determines the relative pose of the first and last keyframes based on the rotation matrix R and translation matrix T.

[0154] In this embodiment of the application, the processor can obtain the three-dimensional spatial points of each key frame in the initial key frame based on triangulation calculation.

[0155] For example, the processor can perform triangulation calculations based on the relative poses of the first and last keyframes to obtain the three-dimensional spatial points of the first and last keyframes. Then, the processor can determine the three-dimensional spatial points of each keyframe remaining in the initial keyframes (excluding the first and last keyframes) based on the three-dimensional spatial points of the first and last keyframes and the feature matching relationships between frames in the initial keyframes, thereby obtaining the three-dimensional spatial points of each keyframe in the initial keyframes.

[0156] Here, the above-mentioned processor performing triangulation calculations may include the following steps:

[0157] For example, the homogeneous coordinates of a point in three-dimensional space are [x, y, z, 1]. T The projection of a point in three-dimensional space onto an image

[0158]

[0159] Where k is the camera intrinsic parameter matrix, R is the rotation matrix, and T is the translation matrix. Here, parameter P represents k.<R|T> u represents X represents Therefore, we get:

[0160] λu=PX

[0161] Multiplying the difference between both sides by u, we get:

[0162] u^PX=0

[0163] Expanding, we get:

[0164]

[0165] Further results were obtained:

[0166]

[0167] Two of the three equations above are linearly independent because (1) × (-u) - (2) × v = (3), where Pi is a row of matrix P. One frame can form two equations, so two frames can form four equations:

[0168]

[0169] This can be solved using singular value decomposition, where the homogeneous coordinate X is the singular vector of the minimum singular value of H.

[0170] In addition, after obtaining the relative poses of the first keyframe and the last keyframe, and the three-dimensional spatial points of each keyframe in the initial keyframe, the processor can determine the position of the three-dimensional spatial points of each keyframe in the multiple keyframes by projecting them onto the first keyframe and the last keyframe. Then, based on the position obtained by the projection, a projection error is determined, and based on the projection error and the three-dimensional spatial points of each keyframe in the multiple keyframes, the relative poses of each keyframe in the multiple keyframes are determined.

[0171] In addition to determining the reprojection error, the processor can further determine the relative reprojection error. Based on the reprojection error and the three-dimensional spatial points of each key frame in the multiple key frames, the relative pose of each key frame in the multiple key frames is determined, thereby improving the pose estimation accuracy.

[0172] For example, when the processor determines the relative pose of each key frame in the plurality of key frames based on the reprojection error and / or relative reprojection error, it can construct a local optimization problem. The optimization problem uses the reprojection error and / or relative reprojection error as a loss function. When the loss function value reaches a preset error threshold, the relative pose of each key frame in the initial key frame is obtained, thereby improving the accuracy of synchronous localization and mapping initialization.

[0173] Here, before the processor builds the initial map based on the 3D spatial points of each keyframe in the multiple keyframes and the relative poses of each keyframe in the multiple keyframes, it can determine another projection error based on the 3D spatial points of each keyframe in the multiple keyframes. Then, it performs global optimization based on this projection error to obtain the optimized 3D spatial points of each keyframe in the multiple keyframes and the relative poses of each keyframe in the multiple keyframes. Thus, the initial map is built based on the optimized 3D spatial points of each keyframe in the initial keyframes and the relative poses of each keyframe in the initial keyframes.

[0174] In addition to determining the reprojection error, the processor can further determine the relative reprojection error, and then perform global optimization based on the relative reprojection error to obtain the three-dimensional spatial points of each key frame in the above-mentioned multiple key frames and the relative pose of each key frame in the above-mentioned multiple key frames, so as to better perform spatial positioning.

[0175] For example, after determining the reprojection error and relative reprojection error, the processor can construct a global optimization problem. This optimization problem uses the reprojection error and / or relative reprojection error as a loss function. Then, based on this optimization problem, the 3D spatial points of each keyframe in the optimized initial keyframe and the relative pose of each keyframe in the initial keyframe are obtained. Thus, based on the optimized information, an initial map is built, and map point information is accurately provided.

[0176] S502: Determine the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the above synchronous positioning and mapping system.

[0177] S503: Determine the reprojection error based on the above target three-dimensional spatial points.

[0178] S504: Based on the above reprojection error and the maximum parallax corresponding to the above target three-dimensional space point after preprocessing, determine the relative reprojection error. This preprocessing includes the operation of removing the rotation effect.

[0179] S505: Based on the aforementioned relative reprojection error, perform backend optimization on the aforementioned synchronous positioning and mapping system.

[0180] The implementation of steps S502-S505 is the same as that of steps S201-S204 above, and will not be repeated here.

[0181] The processor described in this embodiment can also initialize the synchronous localization and mapping system, thereby ensuring that subsequent processing proceeds normally. Furthermore, the processor uses the parallax to weight the reprojection error, obtaining a relative reprojection error. This relative reprojection error describes the difference between the spatial position of a point and the spatial position of the camera, thus providing a more robust residual construction for the optimization problem of synchronous localization and mapping, enabling better spatial localization. This solves the problem that existing backend optimizations cannot adequately represent the relationship between the reprojection error and the depth of the point, lacking spatial information description, which may lead to low accuracy in the optimized pose estimation.

[0182] In the embodiments of this application, such as Figure 6 As shown, the processor can first initialize the synchronous localization and mapping system, and then determine the target 3D spatial point among the 3D spatial points of multiple keyframes located by the synchronous localization and mapping system. Furthermore, the processor can determine the reprojection error based on the target 3D spatial point, and determine the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the target 3D spatial point after preprocessing. This preprocessing includes removing the effects of rotation. Specifically, the processor determines the maximum disparity after removing rotation, calculates the ratio of the reprojection error to the maximum disparity, and uses this ratio as the relative reprojection error. Finally, based on the relative reprojection error, the processor performs backend optimization of the synchronous localization and mapping system. Alternatively, both the relative reprojection error and the reprojection error can be considered simultaneously for backend optimization of the synchronous localization and mapping system.

[0183] Compared with existing simultaneous localization and mapping (SLT) backend optimization, the processor described above uses the parallax to provide weights for the reprojection error, thereby obtaining the relative reprojection error. This relative reprojection error describes the difference between the spatial position of a point and the spatial position of the camera, thus providing a more robust residual construction for the optimization problem of SLT and mapping, and better performing spatial localization. This solves the problem that existing backend optimization cannot well represent the relationship between the reprojection error and the depth of the point, lacks spatial information description, and may lead to low accuracy of the optimized pose estimation.

[0184] Corresponding to the synchronous positioning and mapping backend optimization method in the above embodiment, Figure 7 This is a schematic diagram of the synchronous positioning and mapping backend optimization device provided in an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. Figure 7This is a schematic diagram of a synchronous positioning and mapping backend optimization device provided in an embodiment of this application. The synchronous positioning and mapping backend optimization device 70 includes: a spatial point determination module 701, a reprojection error determination module 702, a relative reprojection error determination module 703, and a backend optimization module 704. The synchronous positioning and mapping backend optimization device can be the processor itself, or a chip or integrated circuit that implements the processor's functions. It should be noted that the division of the spatial point determination module, reprojection error determination module, relative reprojection error determination module, and backend optimization module is only a logical functional division; physically, they can be integrated or independent.

[0185] Among them, the spatial point determination module 701 is used to determine the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system.

[0186] The reprojection error determination module 701 is used to determine the reprojection error based on the target three-dimensional spatial points.

[0187] The relative reprojection error determination module 703 is used to determine the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the target three-dimensional space point after preprocessing. The preprocessing includes the operation of removing the rotation effect.

[0188] The synchronous positioning and mapping system undergoes backend optimization. The backend optimization module 704 is used to optimize the system based on the relative reprojection error.

[0189] In one possible implementation, the relative reprojection error determination module 703 is specifically used for:

[0190] Calculate the ratio of the reprojection error to the maximum parallax;

[0191] The ratio is taken as the relative reprojection error.

[0192] In one possible implementation, the reprojection error determination module 702 is specifically used for:

[0193] Determine the position obtained by projecting the target three-dimensional spatial point;

[0194] The reprojection error is determined based on the position obtained from the projection.

[0195] In one possible implementation, the backend optimization module 704 is specifically used for:

[0196] Determine whether the relative reprojection error reaches a preset error threshold;

[0197] The spatial point determination module 701 re-executes the step of determining the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located in the synchronous positioning and mapping system, so that the relative reprojection error reaches the preset error threshold. The back-end optimization module 704 performs back-end optimization on the synchronous positioning and mapping system based on the relative reprojection error. If the relative reprojection error is not minimized, the relative reprojection error is minimized to obtain the optimized system.

[0198] The apparatus provided in this application embodiment can be used to perform the above-described... Figure 2 The technical solutions of the method embodiments are similar in implementation principle and technical effect, and will not be described again here.

[0199] Figure 8 This is a schematic diagram of another synchronous positioning and mapping backend optimization device provided in an embodiment of this application. Figure 7 Based on the embodiment shown, the above-mentioned synchronous positioning and mapping backend optimization device 70 further includes: an error determination module 705.

[0200] The error determination module 705 is used to determine multiple target keyframes corresponding to the target three-dimensional space point before the relative reprojection error determination module 703 determines the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the target three-dimensional space point after preprocessing.

[0201] Based on the key point coordinates of key frame ui, key point coordinates of key frame uj, intrinsic parameter matrix of image acquisition device in the synchronous positioning and mapping system, and rotation matrix from key frame uj to key frame ui, multiple disparities corresponding to the target 3D space point after preprocessing are determined, wherein key frame ui and key frame uj are any two key frames among the multiple target key frames, i = 1, 2, ... m, j = 1, 2, ... m, and m is equal to the number of the multiple target key frames;

[0202] Among multiple disparities corresponding to a point in three-dimensional space, the maximum disparity is determined. This is based on the reprojection error and...

[0203] In one possible implementation, the error determination module 705 is specifically used for:

[0204] Calculate the product of the intrinsic parameter matrix, the rotation matrix, the inverse of the intrinsic parameter matrix, and the keypoint coordinates of the keyframe uj;

[0205] The difference between the keypoint coordinates of the keyframe UI and the product is used to determine multiple disparities corresponding to the target 3D space point after preprocessing. The relative reprojection error is determined based on the reprojection error and the maximum disparity.

[0206] The apparatus provided in this application embodiment can be used to perform the above-described... Figure 4 The technical solutions of the method embodiments are similar in implementation principle and technical effect, and will not be described again here.

[0207] Figure 9 This is a schematic diagram of another synchronous positioning and mapping backend optimization device provided in an embodiment of this application. Figure 7 Based on the embodiment shown, the above-mentioned synchronous positioning and mapping backend optimization device 70 further includes: synchronous positioning and mapping initialization module 706.

[0208] The synchronous positioning and mapping initialization module 706 is used to initialize the synchronous positioning and mapping system before the spatial point determination module 701 determines the target three-dimensional spatial point from the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system.

[0209] In one possible implementation, the synchronous positioning and mapping initialization module 706 is specifically used for:

[0210] Acquire a preset number of consecutive frame images, and perform the preprocessing on the preset number of consecutive frame images;

[0211] Using a pre-built, adaptively sized sliding window, initial keyframes are selected from a preset number of consecutive frame images after removing the effects of rotation. The initial keyframes include multiple keyframes.

[0212] Synchronous localization and mapping initialization. An initial map is built based on the 3D spatial points of each keyframe in the initial keyframe and the relative poses of each keyframe in the initial keyframe.

[0213] In one possible implementation, the synchronous positioning and mapping initialization module 706 is specifically used for:

[0214] Determine the relative pose of the first keyframe and the last keyframe among the plurality of keyframes;

[0215] Based on the relative pose of the first keyframe and the last keyframe, obtain the three-dimensional spatial points of each keyframe in the plurality of keyframes.

[0216] The relative pose of each key frame in the plurality of key frames is determined based on the relative pose of the first key frame and the last key frame, and the three-dimensional spatial points of each key frame in the plurality of key frames.

[0217] Based on the determined reprojection error, the relative pose of each keyframe in the initial keyframe is determined.

[0218] In one possible implementation, the synchronous positioning and mapping initialization module 706 is specifically used for:

[0219] The three-dimensional spatial points of each key frame in the plurality of key frames are determined, and their positions are obtained by projecting them onto the first key frame and the last key frame;

[0220] Based on the position obtained from the projection, a first projection error is determined;

[0221] Based on the reprojection error and the three-dimensional spatial points of each keyframe in the plurality of keyframes, the relative pose of each keyframe in the plurality of keyframes is determined.

[0222] In one possible implementation, the synchronous positioning and mapping initialization module 706 is specifically used for:

[0223] Based on the three-dimensional spatial points of each keyframe in the plurality of keyframes, another projection error is determined;

[0224] Global optimization is performed based on the reprojection error to obtain the three-dimensional spatial points of each key frame in the multiple key frames and the relative pose of each key frame in the multiple key frames after optimization.

[0225] The initial map is established based on the 3D spatial points of each keyframe in the optimized initial keyframe and the relative poses of each keyframe in the initial keyframe.

[0226] The apparatus provided in this application embodiment can be used to perform the above-described... Figure 5 The technical solutions of the method embodiments are similar in implementation principle and technical effect, and will not be described again here.

[0227] Optionally, Figure 10A , 10B The diagram above (10C) schematically illustrates a possible basic hardware architecture of the synchronous positioning and mapping backend optimization device described in this application.

[0228] See Figure 10A , 10B The synchronous positioning and mapping backend optimization device 1000 includes at least one processor 1001 and a communication interface 1003. Optionally, it may also include a memory 1002 and a bus 1004.

[0229] In the synchronous positioning and mapping backend optimization device 1000, the number of processors 1001 can be one or more. Figure 10A , 10BThe diagram 1001 illustrates only one processor 1001. Optionally, processor 1001 can be a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). If the synchronous positioning and mapping back-end optimization device 1000 has multiple processors 1001, the types of the multiple processors 1001 can be different or the same. Optionally, the multiple processors 1001 of the synchronous positioning and mapping back-end optimization device 1000 can also be integrated into a multi-core processor.

[0230] The memory 1002 stores computer instructions and data; the memory 1002 may store computer instructions and data required to implement the synchronous positioning and mapping back-end optimization method provided in this application, for example, the memory 1002 stores instructions for implementing the steps of the synchronous positioning and mapping back-end optimization method. The memory 1002 may be any one or any combination of the following storage media: non-volatile memory (e.g., read-only memory (ROM), solid-state drive (SSD), hard disk drive (HDD), optical disk), volatile memory.

[0231] The communication interface 1003 can provide information input / output for the at least one processor. It may also include any one or any combination of the following devices: a network interface (e.g., an Ethernet interface), a wireless network card, or other devices with network access capabilities.

[0232] Optionally, the communication interface 1003 can also be used to synchronize the positioning and mapping backend optimization device 1000 with other computing devices or terminals for data communication.

[0233] Further optional, Figure 10A , 10B The bus 1004 is represented by a thick line. The bus 1004 connects the processor 1001 to the memory 1002 and the communication interface 1003. In this way, through the bus 1004, the processor 1001 can access the memory 1002 and can also use the communication interface 1003 to interact with other computing devices or terminals.

[0234] In this application, the synchronous positioning and mapping backend optimization device 1000 executes computer instructions in the memory 1002, causing the synchronous positioning and mapping backend optimization device 1000 to implement the synchronous positioning and mapping backend optimization method provided in this application, or causing the synchronous positioning and mapping backend optimization device 1000 to deploy the synchronous positioning and mapping backend optimization apparatus described above.

[0235] From the perspective of logical functional division, for example, such as Figure 10A As shown, the memory 1002 may include a spatial point determination module 701, a reprojection error determination module 702, a relative reprojection error determination module 703, and a back-end optimization module 704. This inclusion refers only to the fact that the instructions stored in the memory, when executed, can respectively implement the functions of the spatial point determination module, the reprojection error determination module, the relative reprojection error determination module, and the back-end optimization module, and is not limited to the physical structure.

[0236] For example, such as Figure 10B As shown, the memory 1002 may also include an error determination module 705. This inclusion refers only to the ability of the memory to perform the function of the error determination module when the instructions stored in the memory are executed, and is not limited to the physical structure.

[0237] For example, such as Figure 10C As shown, the memory 1002 may also include a synchronous positioning and mapping initialization module 706. This inclusion refers only to the ability of the synchronous positioning and mapping initialization module to function when the instructions stored in the memory are executed, and is not limited to the physical structure.

[0238] In addition, the aforementioned synchronous positioning and mapping backend optimization equipment can, in addition to the above... Figure 10A , 10B Besides being implemented in software, the 10C can also be implemented in hardware as a hardware module or as a circuit unit.

[0239] This application provides a computer-readable storage medium, wherein the computer program product includes computer instructions that instruct a computing device to execute the aforementioned synchronous positioning and mapping backend optimization method provided in this application.

[0240] This application provides a computer program product including computer instructions, which are executed by a processor according to the method described in the first aspect.

[0241] This application provides a chip including at least one processor and a communication interface, wherein the communication interface provides information input and / or output to the at least one processor. Furthermore, the chip may also include at least one memory for storing computer instructions. The at least one processor is used to invoke and execute the computer instructions to perform the synchronous positioning and mapping backend optimization method provided in this application.

[0242] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0243] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0244] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

Claims

1. A method for optimizing the backend of simultaneous localization and mapping, characterized in that, include: Determine the target three-dimensional spatial point from the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system; The multiple keyframes are obtained by preprocessing a preset number of consecutive frame images to remove the effects of rotation, and then filtering based on the preprocessed frame images using a pre-built adaptive-size sliding window. Based on the target three-dimensional spatial points, determine the reprojection error; The relative reprojection error is determined based on the reprojection error and the maximum disparity corresponding to the target three-dimensional space point after preprocessing. The preprocessing includes the operation of removing the rotation effect. Based on the relative reprojection error, the synchronous positioning and mapping system is optimized at the back end.

2. The method according to claim 1, characterized in that, Before determining the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the preprocessed target 3D spatial point, the method further includes: Determine multiple target keyframes corresponding to the target's three-dimensional spatial points; Based on the key point coordinates of key frame ui, key point coordinates of key frame uj, intrinsic parameter matrix of image acquisition device in the synchronous positioning and mapping system, and rotation matrix from key frame uj to key frame ui, multiple disparities corresponding to the preprocessed target 3D spatial points are determined, wherein key frame ui and key frame uj are any two key frames among the multiple target key frames, i=1,2,…m, j=1,2,…m, and m is equal to the number of the multiple target key frames; The maximum disparity is determined from the multiple disparities corresponding to the target three-dimensional space point after the preprocessing.

3. The method according to claim 2, characterized in that, The step of determining multiple disparities corresponding to the preprocessed target 3D spatial points based on the keypoint coordinates of keyframe ui, keypoint coordinates of keyframe uj, the intrinsic parameter matrix of the image acquisition device in the synchronous positioning and mapping system, and the rotation matrix from keyframe uj to keyframe ui includes: Calculate the product of the intrinsic parameter matrix, the rotation matrix, the inverse of the intrinsic parameter matrix, and the keypoint coordinates of the keyframe uj; Based on the difference between the keypoint coordinates of the keyframe UI and the product, multiple disparities corresponding to the target 3D spatial points after preprocessing are determined.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the preprocessed target 3D spatial point includes: Calculate the ratio of the reprojection error to the maximum parallax; The ratio is taken as the relative reprojection error.

5. The method according to any one of claims 1 to 3, characterized in that, The step of determining the reprojection error based on the target three-dimensional space points includes: Determine the position obtained by projecting the target three-dimensional spatial point; The reprojection error is determined based on the position obtained from the projection.

6. The method according to any one of claims 1 to 3, characterized in that, The backend optimization of the synchronous positioning and mapping system based on the relative reprojection error includes: Determine whether the relative reprojection error reaches a preset error threshold; If the relative reprojection error does not reach the preset error threshold, the step of determining the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system is re-executed so that the relative reprojection error reaches the preset error threshold. Based on the relative reprojection error, the backend optimization of the synchronous positioning and mapping system is performed.

7. The method according to any one of claims 1 to 3, characterized in that, Before determining the target 3D spatial point among the 3D spatial points of multiple keyframes located in the synchronous positioning and mapping system, the method further includes: The synchronous positioning and mapping system is initialized.

8. The method according to claim 7, characterized in that, The initialization of the synchronous positioning and mapping system includes: Acquire a preset number of consecutive frame images, and perform the preprocessing on the preset number of consecutive frame images; Using a pre-built, adaptively sized sliding window, initial keyframes are selected from a preset number of consecutive frame images after removing the effects of rotation. The initial keyframes include multiple keyframes. Based on the aforementioned keyframes, synchronous positioning and mapping initialization are performed.

9. The method according to claim 8, characterized in that, The synchronous localization and mapping initialization based on the multiple keyframes includes: Determine the relative pose of the first keyframe and the last keyframe among the plurality of keyframes; Based on the relative pose of the first keyframe and the last keyframe, obtain the three-dimensional spatial points of each keyframe in the plurality of keyframes. The relative pose of each key frame in the plurality of key frames is determined based on the relative pose of the first key frame and the last key frame, and the three-dimensional spatial points of each key frame in the plurality of key frames. An initial map is established based on the three-dimensional spatial points of each keyframe and the relative poses of each keyframe.

10. A synchronous positioning and mapping backend optimization device, characterized in that, include: The spatial point determination module is used to determine the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system. The multiple keyframes are obtained by preprocessing a preset number of consecutive frame images to remove the effects of rotation, and then filtering based on the preprocessed frame images using a pre-built adaptive-size sliding window. The reprojection error determination module is used to determine the reprojection error based on the target three-dimensional spatial points; The relative reprojection error determination module is used to determine the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the target three-dimensional space point after preprocessing. The preprocessing includes the operation of removing the rotation effect. The backend optimization module is used to perform backend optimization on the synchronous positioning and mapping system based on the relative reprojection error.

11. The apparatus according to claim 10, characterized in that, It also includes an error determination module, which is used to determine multiple target keyframes corresponding to the target three-dimensional space point before the relative reprojection error determination module determines the relative reprojection error based on the reprojection error and the maximum disparity corresponding to the target three-dimensional space point after preprocessing. Based on the key point coordinates of key frame ui, key point coordinates of key frame uj, intrinsic parameter matrix of image acquisition device in the synchronous positioning and mapping system, and rotation matrix from key frame uj to key frame ui, multiple disparities corresponding to the preprocessed target 3D spatial points are determined, wherein key frame ui and key frame uj are any two key frames among the multiple target key frames, i=1,2,…m, j=1,2,…m, and m is equal to the number of the multiple target key frames; The maximum disparity is determined from the multiple disparities corresponding to the target three-dimensional space point after the preprocessing.

12. The apparatus according to claim 11, characterized in that, The error determination module is specifically used for: Calculate the product of the intrinsic parameter matrix, the rotation matrix, the inverse of the intrinsic parameter matrix, and the keypoint coordinates of the keyframe uj; Based on the difference between the keypoint coordinates of the keyframe UI and the product, multiple disparities corresponding to the target 3D spatial points after preprocessing are determined.

13. The apparatus according to any one of claims 10 to 12, characterized in that, The relative reprojection error determination module is specifically used for: Calculate the ratio of the reprojection error to the maximum parallax; The ratio is taken as the relative reprojection error.

14. The apparatus according to any one of claims 10 to 12, characterized in that, The reprojection error determination module is specifically used for: Determine the position obtained by projecting the target three-dimensional spatial point; The reprojection error is determined based on the position obtained from the projection.

15. The apparatus according to any one of claims 10 to 12, characterized in that, The backend optimization module is specifically used for: Determine whether the relative reprojection error reaches a preset error threshold; If the relative reprojection error does not reach the preset error threshold, the spatial point determination module re-executes the step of determining the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located in the synchronous positioning and mapping system, so that the relative reprojection error reaches the preset error threshold. The back-end optimization module performs back-end optimization on the synchronous positioning and mapping system based on the relative reprojection error.

16. The apparatus according to any one of claims 10 to 12, characterized in that, It also includes a synchronous positioning and mapping initialization module, which is used to initialize the synchronous positioning and mapping system before the spatial point determination module determines the target three-dimensional spatial point among the three-dimensional spatial points of multiple key frames located by the synchronous positioning and mapping system.

17. The apparatus according to claim 16, characterized in that, The synchronous positioning and mapping initialization module is specifically used for: Acquire a preset number of consecutive frame images, and perform the preprocessing on the preset number of consecutive frame images; Using a pre-built, adaptively sized sliding window, initial keyframes are selected from a preset number of consecutive frame images after removing the effects of rotation. The initial keyframes include multiple keyframes. Based on the aforementioned keyframes, synchronous positioning and mapping initialization are performed.

18. A synchronous positioning and mapping backend optimization device, characterized in that, include: processor; Memory; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor, the computer program including instructions for performing the method as described in any one of claims 1-9.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that causes the server to perform the method according to any one of claims 1-9.

20. A computer program product, characterized in that, Includes computer instructions, which are executed by a processor according to any one of claims 1-9.

Images