Map construction method and device based on multi-camera module

Abstract

Embodiments of the present application disclose a kind of map construction method and device based on multi-camera module.The map construction method based on multi-camera module includes: obtaining the first image information of space-time synchronous camera and the second image information of space-time asynchronous camera;Obtain the key frame of main camera and the key frame of auxiliary camera in the first image information, and the key frame queue of space-time asynchronous camera in the second image information;According to the key frame of main camera and the key frame of auxiliary camera, generate first key frame set, and according to the key frame queue in the second image information, generate second key frame set;According to first key frame set and second key frame set, carry out sparse mapping, obtain initial map;According to initial map, carry out dense mapping, obtain target map.The scheme provided by the present application can use multi-camera module to carry out the technical effect of pure vision three-dimensional reconstruction.

Description

Technical Field

[0001] This invention relates to the field of wearable device technology applications, and in particular to a map building method and apparatus based on a multi-camera module. Background Technology

[0002] AR and VR technologies can be used in fields such as AR / VR glasses, robots, and autonomous vehicles. The visual maps used in scenarios where AR / VR devices are used are usually constructed by capturing images using multiple camera modules.

[0003] In related technologies, when constructing maps for AR / VR technology, commonly used multiple camera modules include: a binocular grayscale camera and at least one RGB camera, or a quad grayscale camera module and at least one RGB camera; when performing pure vision-based 3D reconstruction through multiple camera modules, the clock chips of each camera are different, so even if the images are captured at the same time, the images cannot be synchronized.

[0004] There is currently no effective solution to the problem of image asynchrony caused by clock discrepancies when multiple camera modules are performing purely visual 3D reconstruction. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention aims to provide a map construction method and apparatus based on multiple camera modules, thereby at least resolving the issue of image asynchrony caused by clock discrepancies when multiple camera modules perform purely visual 3D reconstruction.

[0006] The technical solution of this invention is implemented as follows: In a first aspect, embodiments of the present invention provide a map construction method based on a multi-camera module, comprising: acquiring first image information from a spatiotemporally synchronized camera and second image information from a spatiotemporally asynchronous camera; acquiring keyframes from a main camera and keyframes from an auxiliary camera in the first image information, and a keyframe queue from the spatiotemporally asynchronous camera in the second image information; generating a first keyframe set based on the keyframes from the main camera and the auxiliary camera, and generating a second keyframe set based on the keyframe queue from the second image information; performing sparse mapping based on the first keyframe set and the second keyframe set to obtain an initial map; and performing dense mapping based on the initial map to obtain a target map.

[0007] Optionally, acquiring the first image information of the spatiotemporally synchronized camera and the second image information of the spatiotemporally asynchronous camera includes: acquiring the pose of the spatiotemporally synchronized camera through a calibration module; acquiring the synchronization information between the timestamp, image, and pose of the spatiotemporally synchronized camera through a positioning and tracking module; acquiring the clock synchronization information between the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera through a time calibration module; generating the first image information based on the pose, timestamp, synchronization information between the image and pose, and clock synchronization information; acquiring the pose of the spatiotemporally asynchronous camera through a calibration module; acquiring the synchronization information between the timestamp and image of the spatiotemporally asynchronous camera through a positioning and tracking module; generating the second image information based on the synchronization information between the pose, timestamp, and image of the spatiotemporally asynchronous camera, and clock synchronization information; wherein, the clock synchronization information is used to calibrate the timestamps of the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera to the same clock.

[0008] Optionally, obtaining the keyframe queue of the spatiotemporally asynchronous camera in the second image information includes: obtaining keyframes of the spatiotemporally asynchronous camera from the second image information according to a third preset keyframe filtering strategy; wherein, the third preset keyframe filtering strategy includes: obtaining keyframes according to the target purpose through a preset application strategy; obtaining keyframes at the same time as obtaining keyframes of the spatiotemporally synchronous camera according to a preset correlation strategy; obtaining keyframes within a specified continuous time period according to a preset continuity strategy; and generating a keyframe queue based on the filtered keyframes.

[0009] Further, optionally, the method further includes: when the spatiotemporal asynchronous camera selects a new keyframe according to a preset application strategy or a preset continuity strategy, if the continuous time or continuous frames do not conform to the preset correlation strategy and a keyframe cannot be inserted, then it is determined that the continuous time or the continuous frames are greater than a threshold of a specified judgment strategy. Further, optionally, generating a second keyframe set based on the keyframe queue in the second image information includes: comparing the keyframes in the keyframe queue in the second image information with the capture times of the keyframes of the main camera and the keyframes of the auxiliary camera, and filtering out preliminary keyframes whose capture time is less than a preset time interval threshold for each keyframe of the main camera and the keyframe of the auxiliary camera; if multiple preliminary keyframes of the spatiotemporal asynchronous camera satisfy the conditions for the same keyframe of the main camera and the keyframe of the auxiliary camera, then retaining the keyframe closest to the capture time of the keyframe of the main camera and the keyframe of the auxiliary camera, and generating a second keyframe set.

[0010] Optionally, sparse mapping based on the first keyframe set and the second keyframe set to obtain the initial map includes: using the trajectory prior of the spatiotemporally synchronized camera in the first keyframe set, and calculating the pose of the keyframes of the auxiliary camera based on the pose of the main camera and the relative pose of the auxiliary camera to the main camera; matching feature points between the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera based on the pose of the keyframes, registering the prior trajectory of the spatiotemporally asynchronous camera, removing outliers by using the relative pose of the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera as constraints, and performing global optimization to obtain the initial map.

[0011] Further, optionally, the method also includes: calculating the current pose of the main camera based on the pose of the spatiotemporally asynchronous camera and the relative pose between the spatiotemporally asynchronous camera and the main camera; if the difference between the current pose of the main camera and the pose obtained by initial interpolation is greater than a preset value, it is marked as suspicious, and the image of the corresponding spatiotemporally asynchronous camera and the associated 3D point map information are discarded; monitoring the movement speed of adjacent frames in the image of the spatiotemporally asynchronous camera, if it is greater than a preset running speed, the image of the corresponding spatiotemporally asynchronous camera and the associated 3D point map information are discarded; optimizing all 3D point map information and camera poses to determine the relative pose between the spatiotemporally asynchronous camera and the main camera.

[0012] Further, optionally, the prior trajectory registration for the spatiotemporally asynchronous camera includes: acquiring images of the main camera adjacent to the spatiotemporally asynchronous camera before and after the image capture time; interpolating the pose of the main camera at the time of image capture based on the poses of the two main camera images; and calculating an approximate current pose of the spatiotemporally asynchronous camera based on the relative poses between the spatiotemporally asynchronous camera and the spatiotemporally synchronous camera, which is then used as the prior pose of the spatiotemporally asynchronous camera for mapping.

[0013] Optionally, dense mapping is performed based on the initial map to obtain the target map, including: querying the poses between spatiotemporally synchronized cameras and spatiotemporally asynchronous cameras and performing stereo matching; and coloring the point cloud model based on the stereo matching to obtain the target map.

[0014] Optionally, the method also includes: using the target map to perform data augmentation on the capabilities of the spatiotemporally asynchronous camera.

[0015] Secondly, embodiments of the present invention provide a map building apparatus based on a multi-camera module, comprising: a first acquisition module for acquiring first image information from a spatiotemporally synchronized camera and second image information from a spatiotemporally asynchronous camera; a second acquisition module for acquiring keyframes from a main camera and a secondary camera in the first image information, and a keyframe queue from the spatiotemporally asynchronous camera in the second image information; a queue generation module for generating a first keyframe set based on the keyframes from the main camera and the secondary camera, and generating a second keyframe set based on the keyframe queue in the second image information; a first mapping module for performing sparse mapping based on the first keyframe set and the second keyframe set to obtain an initial map; and a second mapping module for performing dense mapping based on the initial map to obtain a target map.

[0016] This invention provides a map construction method and apparatus based on a multi-camera module. It acquires first image information from a spatiotemporally synchronized camera and second image information from a spatiotemporally asynchronous camera; acquires keyframes from the main camera and auxiliary camera in the first image information, and a queue of keyframes from the spatiotemporally asynchronous camera in the second image information; generates a first keyframe set based on the keyframes from the main camera and auxiliary camera, and generates a second keyframe set based on the keyframe queue in the second image information; performs sparse mapping based on the first and second keyframe sets to obtain an initial map; and performs dense mapping based on the initial map to obtain a target map, thereby achieving the technical effect of pure visual 3D reconstruction using a multi-camera module. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a map construction method based on a multi-camera module provided in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of module processing in a map construction method based on a multi-camera module provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of a map building device based on a multi-camera module provided in Embodiment 2 of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish different objects, rather than to limit a specific order.

[0020] It should also be noted that the various embodiments of the present invention described below can be executed individually or in combination with each other, and the embodiments of the present invention do not impose specific limitations in this regard.

[0021] Technical terms used in the embodiments of this application: External parameter calibration: refers to the fact that the relative poses between cameras have been obtained through calibration.

[0022] Spacetime synchronization: using the same clock, and external parameters have been calibrated.

[0023] Spatiotemporally synchronized camera set: a set of spatiotemporally synchronized camera modules (usually binocular or multi-lens grayscale cameras).

[0024] Spatiotemporally asynchronous cameras: Their extrinsic parameters are also calibrated, using different clocks. Therefore, when capturing images at the same timestamp, these images, along with those from the camera set, may not actually be captured at the same moment. The time difference may be a static, fixed value or a dynamic value, depending on the hardware triggering method. However, it is certain that after roughly aligning their clocks, the "so-called same moment" images obtained by spatiotemporally asynchronous cameras and spatiotemporally synchronous cameras will generally fall within a certain time interval, with an error typically not exceeding 1 second.

[0025] Main camera: VIO or pose tracking module, the form of the output tracking pose, is generally relative to a certain camera in the camera set, and is called the main camera.

[0026] Auxiliary camera: Other cameras are referred to as auxiliary cameras.

[0027] Example 1 In a first aspect, embodiments of the present invention provide a map construction method based on a multi-camera module. Figure 1 This is a flowchart illustrating a map construction method based on a multi-camera module according to Embodiment 1 of the present invention; as shown below. Figure 1 As shown, the map construction method based on a multi-camera module provided in this application includes: Step S102: Obtain the first image information from the spatiotemporally synchronized camera and the second image information from the spatiotemporally asynchronous camera; Optionally, step S102, obtaining the first image information of the spatiotemporally synchronized camera and the second image information of the spatiotemporally asynchronous camera, includes: obtaining the pose of the spatiotemporally synchronized camera through a calibration module; obtaining synchronization information between the timestamp, image, and pose of the spatiotemporally synchronized camera through a positioning and tracking module; obtaining clock synchronization information between the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera through a time calibration module; generating the first image information based on the pose, timestamp, synchronization information between the image and pose, and clock synchronization information; obtaining the pose of the spatiotemporally asynchronous camera through a calibration module; obtaining synchronization information between the timestamp and image of the spatiotemporally asynchronous camera through a positioning and tracking module; generating the second image information based on the pose, timestamp, and image of the spatiotemporally asynchronous camera, and clock synchronization information; wherein, the clock synchronization information is used to calibrate the timestamps of the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera to the same clock.

[0028] Specifically, such as Figure 2 As shown, Figure 2 This is a schematic diagram of module processing in a map construction method based on a multi-camera module provided in Embodiment 1 of the present invention. Typically, in systems such as AR glasses and robots, the following can be obtained from the calibration module: Pose (external parameters) between spatiotemporally synchronized and spatiotemporally asynchronous cameras. Existing positioning and tracking modules such as SLAM can be used to obtain: Spatiotemporal synchronized camera: timestamps, images, and poses are synchronized; Non-spatiotemporal synchronized camera: timestamps, images; A readily available time calibration module can be used to obtain: The clock synchronization information between spatiotemporally synchronized and spatiotemporally asynchronous cameras is used to roughly correct the timestamps of each camera to the same clock.

[0029] In typical cases, spatiotemporally synchronized cameras are generally grayscale monocular-binocular camera groups, while spatiotemporally asynchronous cameras are generally color, infrared, or other types of cameras or camera groups. Grayscale and color will be used to replace them in the following text.

[0030] Step S104: Obtain the keyframes of the main camera and the auxiliary camera in the first image information, and the keyframe queue of the spatiotemporal asynchronous camera in the second image information. Optionally, step S104, obtaining keyframes from the main camera and the auxiliary camera in the first image information, includes: obtaining keyframes from the main camera in the first image information according to a first preset keyframe filtering strategy, wherein the first preset keyframe filtering strategy includes: obtaining SLAM keyframes, or selecting keyframes based on feature point matching and viewpoint overlap; obtaining keyframes within a specified continuous time period; and obtaining keyframes from the auxiliary camera in the first image information according to a second preset keyframe filtering strategy, wherein the second preset keyframe filtering strategy includes: obtaining keyframes captured at the same time as the main camera.

[0031] Specifically, the keyframe selection strategy of the grayscale main camera (i.e., the main camera in this embodiment) (i.e., the first keyframe selection strategy in this embodiment) is primarily based on meeting the needs of 3D reconstruction, including: SLAM keyframes, or selected based on feature point matching, viewpoint overlap, etc.; It has a certain degree of temporal continuity; A simple addition to the regular filtering strategy is to take one or a group of (multi-camera) keyframes every time the interval exceeds a certain time threshold.

[0032] Keyframe selection for the grayscale auxiliary camera (i.e., the auxiliary camera in this embodiment) (i.e., the second keyframe selection strategy in this embodiment): consistent with the main camera, shooting at the same time.

[0033] Optionally, step S104, obtaining the keyframe queue of the spatiotemporally asynchronous camera in the second image information, includes: obtaining keyframes of the spatiotemporally asynchronous camera from the second image information according to a third preset keyframe filtering strategy; wherein, the third preset keyframe filtering strategy includes: obtaining keyframes according to the target purpose through a preset application strategy; obtaining keyframes at the same time as obtaining keyframes of the spatiotemporally synchronous camera according to a preset correlation strategy; obtaining keyframes within a specified continuous time period according to a preset continuity strategy; and generating a keyframe queue based on the filtered keyframes.

[0034] Specifically, the keyframe strategy selection for the color camera (i.e., the spatiotemporal asynchronous camera provided in this application embodiment) (i.e., the third preset keyframe filtering strategy provided in this application embodiment) must simultaneously meet the following conditions: Strategy one, which prioritizes meeting the end-use objective, is called the application strategy; Strategy 2 requires that the keyframes of grayscale be captured simultaneously or close to each other in terms of capture time; this is called the correlation strategy.

[0035] Strategy three, which has a certain degree of temporal continuity, is called a continuous strategy.

[0036] Specifically, the image format formula is as follows:

[0037] Where the function This indicates whether the frame has been selected as a keyframe; a value of 0 means it is not a keyframe, and a value of 1 means it is a keyframe.

[0038] When all three strategies are satisfied, the frame is selected as a keyframe.

[0039] In the formula: It is the current frame of the spatiotemporally asynchronous camera. It is the latest keyframe from the spatiotemporally asynchronous camera. It is a collection of spatiotemporally asynchronous keyframes. It is an association strategy. One way is to input the current frame of the spatiotemporally asynchronous camera and compare it with the keyframe set of all spatiotemporally synchronous cameras. If they are close, the value is assigned to 1.

[0040] One approach to this strategy is to input the current frame of the spatiotemporally asynchronous camera and the image information of that frame to make a judgment.

[0041] This represents the application strategy judgment function based on the current frame of a spatiotemporally asynchronous camera. For example, if the strategy represented by the image format formula tends to insert keyframes, then... The value is assigned to 1.

[0042] For example, to obtain a map model with color textures, a strategy is usually selected based on the perspective of the textures in the 3D reconstructed model. To obtain a localization map for global localization, the frame selection strategy of the localization map is usually combined with factors such as viewpoint, feature points, and depth of field. To create scene understanding datasets such as gesture recognition, semantic understanding, and object recognition, selection is usually made by combining factors such as viewpoint and screen ratio of the object or region of interest.

[0043] It is a continuous strategy. One way is to input the current frame of the spatiotemporally asynchronous camera and compare it with the latest keyframe added in the keyframe set of the spatiotemporally asynchronous camera. If the time interval exceeds a certain period, the value is assigned to 1.

[0044] Furthermore, the key frame filtering strategy in this application embodiment also includes: when the time-space asynchronous camera selects a new key frame according to a preset application strategy or a preset continuity strategy, if the continuous time or continuous frames do not conform to the preset correlation strategy and a key frame cannot be inserted, then it is determined that the continuous time or the continuous frames are greater than the threshold of the specified judgment strategy.

[0045] Specifically, if a spatiotemporally asynchronous camera needs to select a new keyframe according to an application strategy or a continuity strategy, and the keyframe does not meet the correlation strategy and exceeds a specified threshold, then the spatiotemporally synchronous camera (main camera and auxiliary camera) is triggered to add a new keyframe, so that the frames of the spatiotemporally asynchronous camera meet the correlation strategy within the same time period. The threshold can be the maximum acceptable difference or time interval between keyframes extracted from the application strategy / continuity strategy.

[0046] The keyframe filtering strategy described above can prevent too many keyframes from spatiotemporally asynchronous cameras from being discarded due to the lack of keyframes from the main and auxiliary cameras at similar times, thus avoiding information loss.

[0047] Step S106: Generate a first keyframe set based on the keyframes of the main camera and the keyframes of the auxiliary camera, and generate a second keyframe set based on the keyframe queue in the second image information. Further, optionally, step S106, which generates a second keyframe set based on the keyframe queue in the second image information, includes: comparing the keyframes in the keyframe queue in the second image information with the capture times of the keyframes of the main camera and the keyframes of the auxiliary camera, and selecting the preparatory keyframes whose capture time is less than a preset time interval threshold for each keyframe of the main camera and the keyframe of the auxiliary camera; if multiple preparatory keyframes of spatiotemporally asynchronous cameras meet the conditions for the same keyframe of the main camera and the keyframe of the auxiliary camera, then retaining the keyframe closest to the capture time of the keyframe of the main camera and the keyframe of the auxiliary camera, and generating a second keyframe set.

[0048] A set of preliminary keyframes is prepared and compared with the capture time of grayscale keyframes. Keyframes whose capture time is less than a certain time interval threshold are selected. If multiple color preliminary keyframes meet the conditions for the same grayscale keyframe, the one closest to the capture time of the grayscale keyframe is retained.

[0049] Step S108: Perform sparse mapping based on the first keyframe set and the second keyframe set to obtain the initial map; Optionally, step S108, which involves sparse mapping based on the first keyframe set and the second keyframe set to obtain the initial map, includes: calculating the pose of the keyframes of the auxiliary camera based on the trajectory prior of the spatiotemporally synchronized camera in the first keyframe set, and based on the pose of the main camera and the relative pose of the auxiliary camera to the main camera; matching feature points between the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera based on the pose of the keyframes, registering the prior trajectory of the spatiotemporally asynchronous camera, removing outliers by using the relative pose of the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera as constraints, and performing global optimization to obtain the initial map.

[0050] Further, optionally, the prior trajectory registration for the spatiotemporally asynchronous camera includes: acquiring images of the main camera adjacent to the spatiotemporally asynchronous camera before and after the image capture time; interpolating the pose of the main camera at the time of image capture based on the poses of the two main camera images; and calculating an approximate current pose of the spatiotemporally asynchronous camera based on the relative poses between the spatiotemporally asynchronous camera and the spatiotemporally synchronous camera, which is then used as the prior pose of the spatiotemporally asynchronous camera for mapping.

[0051] Specifically, the grayscale camera group uses trajectory priors to obtain the map: Each keyframe of the main camera has its own pose. Other auxiliary cameras in the spatiotemporally synchronized camera group can use the pose of the main camera and the relative pose (extrinsic parameter) between the auxiliary camera and the main camera to obtain the pose of the corresponding keyframe.

[0052] The 3D reconstruction of the main camera and auxiliary camera can be performed by inputting these initial poses to accelerate the optimization iteration process. The optimization iteration process includes: feature point extraction, matching, using these camera poses to perform triangulation and generate 3D points in space; and continuous BA (Bundle Adjustment) optimization of the system, with the optimization amount being the 3D points and camera poses.

[0053] During the initial 3D reconstruction optimization process, the extrinsic parameters (relative pose) of the main camera and each auxiliary camera are fixed. After several rounds of optimization, once the target error is considered to have reached the expected threshold, several more rounds of optimization are added, incorporating external factors into the optimization process.

[0054] Feature point matching is performed between color (spatial-asynchronous camera) and grayscale camera images, specifically as follows: feature point matching between color and grayscale cameras; The process involves prior trajectory registration for the color camera, finding adjacent main camera images before and after the color camera image capture time, and interpolating the main camera pose at the moment the color camera image was captured based on the poses of the two main camera images. Then, using the extrinsic parameters of both the color camera and the main camera, an approximate pose of the color camera at that moment is calculated and used as the color camera's prior pose for mapping.

[0055] During the initial 3D reconstruction optimization process, no new triangulation operation is performed between the color camera and the grayscale camera; triangulation is only performed between the color cameras. However, if there exists a feature point match between the color camera and the grayscale camera such that the feature point exists in the grayscale camera and has a corresponding 3D point, then the corresponding feature point in the color camera is associated with that 3D point.

[0056] The system is continuously optimized using Body Analyzer (BA), with the optimization parameters being the newly generated 3D points from the color camera and the color camera pose.

[0057] Using the extrinsic parameters of the grayscale and color cameras as constraints to remove outliers, and then performing the following validations using the optimized mapping results: The SLAM algorithm is used to obtain keyframes and their poses, which serve as the initial values ​​for the poses of each keyframe in 3D reconstruction.

[0058] Optionally, the map construction method based on a multi-camera module provided in this application embodiment further includes: calculating the current pose of the main camera based on the pose of the spatiotemporally asynchronous camera and the relative pose between the spatiotemporally asynchronous camera and the main camera; if the difference between the current pose of the main camera and the pose obtained by initial interpolation is greater than a preset value, it is marked as suspicious, and the image of the corresponding spatiotemporally asynchronous camera and its associated 3D point map information are discarded; monitoring the movement speed of adjacent frames in the image of the spatiotemporally asynchronous camera, if it is greater than a preset running speed, the image of the corresponding spatiotemporally asynchronous camera and its associated 3D point map information are discarded. All 3D point map information and camera poses are optimized to determine the relative pose between the spatiotemporally asynchronous camera and the main camera.

[0059] Specifically, this includes, but is not limited to: using the pose obtained by the color camera in the mapping, the extrinsic parameters of the color camera and the main camera, and inversely calculating the pose of the main camera at this moment; if the currently calculated pose of the main camera differs too much from the pose initially obtained by interpolation, it is marked as suspicious, and the corresponding color camera image and associated 3D point map information are discarded; observe the movement speed of adjacent frames of the color camera, and if it exceeds the operating speed that a normal device can achieve (i.e., the preset operating speed in this application embodiment; for example, in the AR glasses usage scenario, it is necessary to verify that the movement speed is not greater than the walking speed of a general user, etc.), then the corresponding color camera image and associated 3D point map information are discarded.

[0060] Joint optimization involves performing a global optimization of the system again, optimizing the poses of all 3D points and cameras, and fixing the extrinsic parameters between cameras.

[0061] Step S110: Perform dense mapping based on the initial map to obtain the target map.

[0062] Optionally, in step S110, dense mapping based on the initial map to obtain the target map includes: querying the poses between the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera, and performing stereo matching; and coloring the point cloud model based on the stereo matching to obtain the target map.

[0063] Specifically, the operations of the stereo matching and depth estimation modules include: First, examine the extrinsic parameters between camera modules to calculate similar camera pairs with a field-of-view overlap exceeding a certain threshold (50%). Then, perform stereo matching on the images corresponding to the keyframe combinations in these camera pairs. Note that "similar" here refers to matching between grayscale images and between color images in a true sense. It should be noted that the 50% threshold is only used as an optimal example for illustration, to implement the map construction method based on multi-camera modules provided in this application embodiment, and is not specifically limited. Then, stereo matching is performed on similar images that do not belong to the same group of keyframes but whose relative poses satisfy the multi-view geometric constraints; where the same group refers to keyframes of the main and auxiliary cameras taken at the same time, and keyframes of spatiotemporally asynchronous cameras taken at close time.

[0064] The point cloud fusion module can obtain dense point clouds using conventional deep fusion and point cloud fusion methods.

[0065] Next, conventional methods such as normal estimation, Poisson, and Delaunay triangulation can be used to obtain the three-dimensional model. Select and use a specific camera source to color the point cloud model; for example, you can use images from a color (non-spatiotemporally calibrated) camera, or its derivatives (such as semantic maps, manually labeled classification maps, etc.), and calculate the model texture based on its pose in the map to color the point cloud model and obtain the final map model.

[0066] Optionally, the map construction method based on a multi-camera module provided in this application embodiment further includes: using a target map to perform data augmentation on the functionality of a spatiotemporally asynchronous camera.

[0067] Specifically, map information is used to enhance the functionality of the spatiotemporal asynchronous camera.

[0068] For example, using a 3D reconstruction model, the color camera pose obtained from the map can be augmented with corresponding feature points. Furthermore, by using the 3D points projected onto the 3D map from the 2D points in that pose, the corresponding 2D-3D point information can be expanded. This facilitates subsequent functions such as relocalization using the color camera.

[0069] For example, a 3D reconstruction model can be used to assist in optimizing the semantic segmentation results of keyframes captured by a color camera.

[0070] Based on the above, in AR glasses using related technologies, the time-synchronized camera module is a grayscale camera. RGB color cameras are rarely time-synchronized. This results in the color camera not participating in the reconstruction, making it impossible to obtain color map model textures.

[0071] The map building method based on a multi-camera module provided in this application uses multiple cameras simultaneously and can produce color models.

[0072] Furthermore, due to the asynchronous clock and image data sources of camera modules in related technologies, their clock and pose information cannot be utilized, and they can generally only participate in traditional 3D reconstruction based on pure image sources. However, traditional 3D reconstruction usually performs poorly in scenes with low texture or repetitive texture, often resulting in mapping failure or producing incorrect maps due to image mismatch.

[0073] The map construction method based on a multi-camera module provided in this application also utilizes information such as the pose of cameras with asynchronous clocks to obtain higher accuracy mapping results.

[0074] This invention provides a map construction method based on a multi-camera module. The method involves acquiring first image information from a spatiotemporally synchronized camera and second image information from a spatiotemporally asynchronous camera; acquiring keyframes from the main camera and auxiliary camera in the first image information, and a queue of keyframes from the spatiotemporally asynchronous camera in the second image information; generating a first keyframe set based on the keyframes from the main camera and auxiliary camera, and generating a second keyframe set based on the keyframe queue in the second image information; performing sparse mapping based on the first and second keyframe sets to obtain an initial map; and performing dense mapping based on the initial map to obtain a target map. This achieves the technical effect of pure visual 3D reconstruction using a multi-camera module.

[0075] Example 2 Secondly, embodiments of the present invention provide a map building device based on a multi-camera module. Figure 3 This is a schematic diagram of a map building device based on a multi-camera module provided in Embodiment 2 of the present invention; as shown Figure 3 As shown, the map building apparatus based on a multi-camera module provided in this application includes: a first acquisition module 30, used to acquire first image information from a spatiotemporally synchronized camera and second image information from a spatiotemporally asynchronous camera; a second acquisition module 32, used to acquire keyframes from the main camera and the auxiliary camera in the first image information, and a keyframe queue from the spatiotemporally asynchronous camera in the second image information; a queue generation module 34, used to generate a first keyframe set based on the keyframes from the main camera and the auxiliary camera, and to generate a second keyframe set based on the keyframe queue in the second image information; a first mapping module 36, used to perform sparse mapping based on the first keyframe set and the second keyframe set to obtain an initial map; and a second mapping module 38, used to perform dense mapping based on the initial map to obtain a target map.

[0076] This invention provides a map building device based on a multi-camera module. It acquires first image information from a spatiotemporally synchronized camera and second image information from a spatiotemporally asynchronous camera; acquires keyframes from the main camera and auxiliary camera in the first image information, and a queue of keyframes from the spatiotemporally asynchronous camera in the second image information; generates a first keyframe set based on the keyframes from the main camera and auxiliary camera, and generates a second keyframe set based on the keyframe queue in the second image information; performs sparse mapping based on the first and second keyframe sets to obtain an initial map; and performs dense mapping based on the initial map to obtain a target map, thereby achieving the technical effect of pure visual 3D reconstruction using a multi-camera module.

[0077] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A map construction method based on a multi-camera module, characterized in that, include: Acquire the first image information from the spatiotemporally synchronized camera and the second image information from the spatiotemporally asynchronous camera; Obtain keyframes from the main camera and the auxiliary camera in the first image information, and the keyframe queue of the spatiotemporal asynchronous camera in the second image information; A first keyframe set is generated based on the keyframes of the main camera and the keyframes of the auxiliary camera, and a second keyframe set is generated based on the keyframe queue in the second image information. Sparse mapping is performed based on the first keyframe set and the second keyframe set to obtain an initial map; Based on the initial map, a dense map is constructed to obtain the target map; The process of acquiring the keyframe queue of the spatiotemporally asynchronous camera in the second image information includes: acquiring keyframes of the spatiotemporally asynchronous camera from the second image information according to a third preset keyframe filtering strategy; wherein, the third preset keyframe filtering strategy includes: acquiring the keyframes according to the target purpose through a preset application strategy; acquiring keyframes at the same time as acquiring keyframes of the spatiotemporally synchronous camera according to a preset correlation strategy; acquiring keyframes within a specified continuous time period according to a preset continuity strategy; and generating the keyframe queue based on the filtered keyframes. The step of generating a second keyframe set based on the keyframe queue in the second image information includes: comparing the keyframes in the keyframe queue in the second image information with the capture times of the keyframes of the main camera and the keyframes of the auxiliary camera, and filtering out the preliminary keyframes whose capture times are less than a preset time interval threshold for each keyframe of the main camera and the keyframe of the auxiliary camera; if multiple preliminary keyframes of the spatiotemporal asynchronous camera meet the conditions for the same keyframe of the main camera and the keyframe of the auxiliary camera, then retaining the keyframe closest to the capture time of the keyframe of the main camera and the keyframe of the auxiliary camera, and generating the second keyframe set; The step of sparse mapping based on the first keyframe set and the second keyframe set to obtain an initial map includes: using the trajectory prior of the spatiotemporally synchronized camera in the first keyframe set, and calculating the pose of the keyframes of the auxiliary camera based on the pose of the main camera and the relative pose of the auxiliary camera to the main camera; matching feature points between the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera based on the pose of the keyframes, registering the prior trajectory of the spatiotemporally asynchronous camera, removing outliers using the relative pose of the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera as constraints, and performing global optimization to obtain the initial map.

2. The map construction method based on a multi-camera module according to claim 1, characterized in that, The acquisition of the first image information from the spatiotemporally synchronized camera and the second image information from the spatiotemporally asynchronous camera includes: The pose of the spatiotemporal synchronized camera is obtained through the calibration module; the synchronization information between the timestamp, image, and pose of the spatiotemporal synchronized camera is obtained through the positioning and tracking module; and the clock synchronization information between the spatiotemporal synchronized camera and the spatiotemporal asynchronous camera is obtained through the time calibration module; the first image information is generated based on the pose, the timestamp, the synchronization information between the image and the pose, and the clock synchronization information. The calibration module obtains the pose of the spatiotemporal asynchronous camera; the positioning and tracking module obtains the synchronization information between the timestamp and the image of the spatiotemporal asynchronous camera; and the second image information is generated based on the pose of the spatiotemporal asynchronous camera, the synchronization information between the timestamp and the image, and the clock synchronization information. The clock synchronization information is used to align the timestamps of the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera to the same clock.

3. The map construction method based on a multi-camera module according to claim 1, characterized in that, The method further includes: When the spatiotemporal asynchronous camera selects a new keyframe according to the preset application strategy or the preset continuity strategy, if the continuous time or continuous frames do not conform to the preset correlation strategy and a keyframe cannot be inserted, then it is determined that the continuous time or the continuous frames are greater than the threshold of the specified judgment strategy.

4. The map construction method based on a multi-camera module according to claim 1, characterized in that, The method further includes: The pose of the current main camera is calculated based on the pose of the spatiotemporal asynchronous camera and the relative pose between the spatiotemporal asynchronous camera and the main camera. If the difference between the current pose of the main camera and the pose obtained by the initial interpolation is greater than a preset value, it is marked as suspicious, and the corresponding spatiotemporal asynchronous camera image and associated 3D point map information are discarded. The movement speed of adjacent frames in the image of the spatiotemporal asynchronous camera is monitored. If it is greater than the preset running speed, the corresponding image of the spatiotemporal asynchronous camera and the associated 3D point map information are discarded. Optimize all 3D point map information and camera poses to determine the relative pose between the spatiotemporal asynchronous camera and the main camera.

5. The map construction method based on a multi-camera module according to claim 1, characterized in that, The prior trajectory registration for the spatiotemporal asynchronous camera includes: The images of the main camera adjacent to the images captured by the spatiotemporal asynchronous camera are acquired. Based on the poses of the two images of the main camera, the pose of the main camera at the time of image capture by the spatiotemporal asynchronous camera is interpolated. Based on the relative poses between the spatiotemporal asynchronous camera and the spatiotemporal synchronous camera, an approximate current pose of the spatiotemporal asynchronous camera is calculated and used as the prior pose of the spatiotemporal asynchronous camera for mapping.

6. The map construction method based on a multi-camera module according to claim 1, characterized in that, The process of dense mapping based on the initial map to obtain the target map includes: Query the pose between the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera, and perform stereo matching; The point cloud model is colored based on the stereo matching to obtain the target map.

7. The map construction method based on a multi-camera module according to claim 1, characterized in that, The method further includes: The target map is used to perform data augmentation on the functionality of the spatiotemporal asynchronous camera.

8. A map building device based on a multi-camera module, characterized in that, include: The first acquisition module is used to acquire the first image information of the spatiotemporally synchronized camera and the second image information of the spatiotemporally asynchronous camera; The second acquisition module is used to acquire key frames of the main camera and key frames of the auxiliary camera in the first image information, as well as the key frame queue of the spatiotemporal asynchronous camera in the second image information. The queue generation module is used to generate a first keyframe set based on the keyframes of the main camera and the keyframes of the auxiliary camera, and to generate a second keyframe set based on the keyframe queue in the second image information. The first mapping module is used to perform sparse mapping based on the first keyframe set and the second keyframe set to obtain an initial map. The second mapping module is used to perform dense mapping based on the initial map to obtain the target map; The process of acquiring the keyframe queue of the spatiotemporally asynchronous camera in the second image information includes: acquiring keyframes of the spatiotemporally asynchronous camera from the second image information according to a third preset keyframe filtering strategy; wherein, the third preset keyframe filtering strategy includes: acquiring the keyframes according to the target purpose through a preset application strategy; acquiring keyframes at the same time as acquiring keyframes of the spatiotemporally synchronous camera according to a preset correlation strategy; acquiring keyframes within a specified continuous time period according to a preset continuity strategy; and generating the keyframe queue based on the filtered keyframes. The step of generating a second keyframe set based on the keyframe queue in the second image information includes: comparing the keyframes in the keyframe queue in the second image information with the capture times of the keyframes of the main camera and the keyframes of the auxiliary camera, and filtering out the preliminary keyframes whose capture times are less than a preset time interval threshold for each keyframe of the main camera and the keyframe of the auxiliary camera; if multiple preliminary keyframes of the spatiotemporal asynchronous camera meet the conditions for the same keyframe of the main camera and the keyframe of the auxiliary camera, then retaining the keyframe closest to the capture time of the keyframe of the main camera and the keyframe of the auxiliary camera, and generating the second keyframe set; The step of sparse mapping based on the first keyframe set and the second keyframe set to obtain an initial map includes: using the trajectory prior of the spatiotemporally synchronized camera in the first keyframe set, and calculating the pose of the keyframes of the auxiliary camera based on the pose of the main camera and the relative pose of the auxiliary camera to the main camera; matching feature points between the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera based on the pose of the keyframes, registering the prior trajectory of the spatiotemporally asynchronous camera, removing outliers using the relative pose of the spatiotemporally synchronized camera and the spatiotemporally asynchronous camera as constraints, and performing global optimization to obtain the initial map.

Images