Map construction method, device, mobile device and computer-readable storage medium

Abstract

The present application is applicable to the field of map construction technology, and provides a map construction method, device, mobile device and computer-readable storage medium, including: relocating the current location of at least one shooting device on a first map to obtain a relocation result; after the relocation result indicates that the current location of at least one shooting device has been successfully relocated, obtaining at least two first image frames; extracting visual feature points from each of the first image frames to obtain corresponding first visual feature points; determining new map points based on each of the first visual feature points; determining the current posture of at least one shooting device; inserting the new map point into the first map based on the new map point and the current posture of at least one shooting device to obtain a second map. Through the above method, the accuracy of the generated map can be improved, and thus the accuracy of positioning can be improved.

Description

Technical Field

[0001] The present application belongs to the field of map construction technology, and in particular, relates to a map construction method, device, mobile device and computer-readable storage medium. Background Art

[0002] Visual sensors can collect information such as texture and color, and because of their low cost, they are usually the first sensors to be considered. For example, the visual simultaneous localization and mapping (SLAM) system uses visual sensors to achieve map construction. Specifically, visual sensors can be used to collect continuous image frames and detect image features from continuous image frames to calculate the movement of the visual sensor and reconstruct the surrounding environment. However, when the environment changes greatly, it is difficult for the visual SLAM system to successfully locate.

[0003] Among the existing methods, deep learning-based methods can be used to improve visual positioning performance in situations where the environment changes greatly. Since deep learning methods rely heavily on high computing power, high computing power is also required when using deep learning-based methods to improve visual positioning performance in situations where the scene changes greatly, resulting in higher costs for products using this method. Summary of the invention

[0004] The embodiments of the present application provide a map construction method, apparatus, mobile device and computer-readable storage medium, which can solve the problem that the visual SLAM system is difficult to successfully locate.

[0005] A map construction method is applied to a mobile device including at least two shooting devices, wherein the shooting ranges of the shooting devices have an intersection and the relative positions between the shooting devices are fixed, and the map construction method comprises:

[0006] Relocating the current location of at least one of the photographing devices on a first map to obtain a relocation result, wherein the first map is a constructed map;

[0007] After the relocation result indicates that the current position of at least one of the shooting devices has been successfully relocated, acquiring at least two first image frames, wherein each of the first image frames is captured by different shooting devices at the same time;

[0008] Extracting visual feature points from each of the first image frames to obtain corresponding first visual feature points;

[0009] Determining a new map point according to each of the first visual feature points;

[0010] Determining a current position of at least one of the camera devices;

[0011] According to the new map point and the current position of at least one of the photographing devices, the new map point is inserted into the first map to obtain a second map.

[0012] In a second aspect, an embodiment of the present application provides a map construction device, which is applied to a mobile device including at least two shooting devices, wherein the shooting ranges of the shooting devices have an intersection, and the relative positions between the shooting devices are fixed, and the map construction device includes:

[0013] a relocation module, configured to relocate the current location of at least one of the photographing devices on a first map to obtain a relocation result, wherein the first map is a constructed map;

[0014] A first image frame acquisition module, configured to acquire at least two first image frames after the relocation result indicates that the current position of at least one of the shooting devices has been successfully relocated, wherein each of the first image frames is captured by different shooting devices at the same time;

[0015] A first visual feature point determination module, used to extract visual feature points from each of the first image frames to obtain corresponding first visual feature points;

[0016] A new map point determination module, used to determine a new map point according to each of the first visual feature points;

[0017] A current posture determination module of a photographing device, used to determine a current posture of at least one of the photographing devices;

[0018] The second map generation module is used to insert the new map point into the first map according to the new map point and the current position of at least one of the photographing devices to obtain a second map.

[0019] In a third aspect, an embodiment of the present application provides a mobile device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a mobile device, enables the mobile device to execute the method described in the first aspect above.

[0022] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0023] In the embodiment of the present application, since the first map is a constructed map, after relocating the shooting device on the first map, the new map point is determined according to the first image frame captured by each shooting device at the same time, which can ensure the accuracy of the coordinates, that is, the accuracy of the new map point can be guaranteed, and thus the accuracy of the generated second map can be improved. In addition, since the current position of the shooting device is also combined when inserting the new map point into the first map, and the current position of the shooting device can reflect the orientation of the shooting device, the position of the new map point can be quickly found in the first map in combination with the orientation of the shooting device, and then the new map point can be quickly inserted into the first map, thereby improving the speed of generating the second map. That is, in the embodiment of the present application, since the second map is generated on the basis of the first map, the generation speed of the second map is improved. At the same time, since the second map is a newly generated map, when the mobile device needs to be positioned, compared with positioning according to the first map, positioning according to the second map can improve the positioning success rate of the mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0025] Figure 1 is a flow chart of a map construction method provided by an embodiment of the present application;

[0026] Figure 2 is a structural schematic diagram of a map construction device provided by an embodiment of the present application;

[0027] Figure 3 It is a structural diagram of a mobile device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0030] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0033] Embodiment 1:

[0034] When using a visual SLAM system for positioning, if the current environment (lighting, objects) is significantly different from the environment when the map was constructed, it will be difficult for the visual SLAM system to successfully locate.

[0035] In order to improve the success rate of positioning using a visual SLAM system, an embodiment of the present application provides a map construction method. In this method, an incremental map (i.e., a second map) is constructed based on a constructed map (i.e., a first map) and a currently acquired image frame. Since the second map includes information about the currently acquired image frame, the information contained in the second map is more accurate than that in the first map, so that when positioning is subsequently performed based on the second map, the corresponding positioning success rate will be higher.

[0036] The map construction method provided in the embodiment of the present application is described below with reference to the accompanying drawings.

[0037] Figure 1A flowchart of a map construction method provided in an embodiment of the present application is shown. The map construction method is applied to a mobile device including at least two shooting devices (i.e., including binocular cameras or multi-eye cameras). The shooting ranges of the shooting devices have an intersection, and the relative positions between the shooting devices are fixed. The map construction method provided in an embodiment of the present application is described in detail as follows:

[0038] Step S11, relocating the current position of at least one of the above-mentioned shooting devices on a first map to obtain a relocation result, wherein the above-mentioned first map is a constructed map.

[0039] Relocating the camera device refers to reconfirming the position or posture of the camera device on the first map. Since the relative positions of the various camera devices on the mobile device are fixed, as long as one of the camera devices is successfully relocated, the other camera devices can be relocated according to their relative positions with the relocated camera device.

[0040] In an embodiment of the present application, if the mobile device is currently in the incremental mapping mode, the shooting device needs to be repositioned. Specifically, the mobile device may consider that it is in the incremental mapping mode after receiving the incremental mapping instruction (such as the user pressing a button of the mobile device), or, if the user pre-sets the time point for incremental mapping, the mobile device considers that it is in the incremental mapping mode after the time point arrives. Alternatively, when the mobile device determines that the brightness of the current scene is significantly different from the brightness when the first map is constructed, it considers that it is in the incremental mapping mode.

[0041] Step S12, after the repositioning result indicates that the current position of at least one of the above-mentioned shooting devices has been successfully repositioned, at least two first image frames are acquired, wherein each of the above-mentioned first image frames is respectively captured by different of the above-mentioned shooting devices at the same time.

[0042] In the embodiment of the present application, the shooting frequencies of each shooting device are the same. For example, it is assumed that the mobile device is equipped with two shooting devices: shooting device 1 and shooting device 2, and the shooting ranges of shooting device 1 and shooting device 2 have an intersection. It is assumed that at time t1, shooting device 1 and shooting device 2 shoot image frame 1 and image frame 2 respectively, and at time t2, they shoot image frame 3 and image frame 4 respectively. After shooting device 1 and shooting device 2 perform shooting actions, the mobile device obtains the image frames shot by shooting device 1 and shooting device 2 at the same time, such as obtaining image frame 1 and image frame 2 as the first image frame. That is, in the present application, each first image frame obtained is a video frame shot by each shooting device at the same time. Since the shooting time is the same and the shooting ranges of the shooting devices have an intersection, the same object must exist in the image frames shot by the shooting devices with the intersection. Therefore, the subsequent mobile device can determine the position information of the object in the world coordinate system according to the position information of the same object in different image frames.

[0043] Step S13: extract visual feature points from each of the first image frames to obtain corresponding first visual feature points.

[0044] Specifically, in order to distinguish the feature points extracted from the first image frame from the feature points of the first map, in the embodiment of the present application, the feature points extracted from each first image frame are referred to as first visual feature points, wherein the number of first visual feature points is usually greater than one.

[0045] In an embodiment of the present application, since the first image frames are captured by at least two different shooting devices, the number of first image frames is greater than 1, that is, in this step, the corresponding first visual feature points and the feature descriptors corresponding to the first visual feature points will be extracted from each first image frame respectively.

[0046] In some embodiments, in order to improve the accuracy and real-time performance of the obtained feature points, the first visual feature point in the embodiment of the present application is an ORB (Oriented FAST and Rotated BRIEF) feature point, and the feature descriptor corresponding to the first visual feature point is a BRISK descriptor.

[0047] Step S14, determining a new map point according to each of the above first visual feature points.

[0048] Specifically, the position of the same location point in different first image frames is determined according to the first visual feature points in each first image frame, and then the coordinates of the location point in the world coordinate system are determined according to the binocular triangulation principle, and finally a new map point is determined according to the location point.

[0049] It should be noted that, since the number of the same location points included in each first image frame is usually greater than 1, the number of new map points obtained is also usually greater than 1.

[0050] Step S15, determining the current position and posture of at least one of the above-mentioned photographing devices.

[0051] Since the relative positions between the various photographing devices are fixed, in the embodiment of the present application, when the current position and posture of one photographing device is determined, the current position and posture of other photographing devices can be determined.

[0052] Step S16: inserting the new map point into the first map based on the new map point and the current position of at least one of the photographing devices to obtain a second map.

[0053] Since the coordinates corresponding to the new map point in the world coordinate system are determined when a new map point is determined, and the current position of the shooting device can reflect the orientation of the shooting device, the new map point can be quickly inserted into the first map in combination with the current position of the shooting device to obtain an incremental map of the first map (i.e., the second map).

[0054] In some embodiments, if the mobile device receives a positioning instruction, it responds to the positioning instruction according to the second map. Since the second map is a newly generated map, positioning according to the second map can improve the success rate of positioning. It should be noted that when incremental mapping needs to be continued later, the second map is used as the constructed map (i.e., the first map) for incremental mapping.

[0055] In an embodiment of the present application, the current position of at least one of the above-mentioned photographing devices is relocated on the first map to obtain a relocation result. After the above-mentioned relocation result indicates that the current position of at least one of the above-mentioned photographing devices has been successfully relocated, at least two first image frames are obtained, and visual feature points are extracted from each of the above-mentioned first image frames to obtain corresponding first visual feature points. Then, new map points are determined based on each of the above-mentioned first visual feature points, and the current position and posture of at least one of the above-mentioned photographing devices are determined. Finally, based on the above-mentioned new map points and the current position and posture of at least one of the above-mentioned photographing devices, the above-mentioned new map points are inserted into the above-mentioned first map to obtain a second map. Since the first map is a constructed map, after relocating the photographing devices on the first map, new map points are determined based on the first image frames respectively taken by each photographing device at the same time, so that the accuracy of the obtained coordinates can be guaranteed, that is, the accuracy of the obtained new map points can be guaranteed, and thus the accuracy of the generated second map can be improved. In addition, since the current position of the shooting device is also combined when inserting a new map point in the first map, and the current position of the shooting device can reflect the orientation of the shooting device, the location of the new map point can be quickly found in the first map in combination with the orientation of the shooting device, and then the new map point can be quickly inserted into the first map, thereby improving the speed of generating the second map. That is, in the embodiment of the present application, since the second map is generated based on the first map, the generation speed of the second map is improved. At the same time, since the second map is a newly generated map, when the mobile device needs to be positioned, compared with positioning based on the first map, positioning based on the second map can improve the positioning success rate of the mobile device.

[0056] In some embodiments, the above step S11 includes:

[0057] A1. Acquire at least two second image frames, wherein each of the second image frames is captured by different capturing devices at the same time.

[0058] The shooting time of the second image frame is different from the shooting time of the first image frame. For example, if the shooting time of the second image frame is T, the shooting time of the first image frame may be (T+1).

[0059] A2. Determine the visual feature points and corresponding feature descriptors of each of the second image frames to obtain second visual feature points and second feature descriptors.

[0060] Specifically, any visual feature point extracted from the second image frame is referred to as a second visual feature point, and a feature descriptor corresponding to the second visual feature point is referred to as a second feature descriptor.

[0061] In some embodiments, in order to improve the accuracy and real-time performance of the obtained feature points, the second visual feature points in the embodiments of the present application are ORB feature points, and the feature descriptors corresponding to the second visual feature points are BRISK descriptors.

[0062] A3. Based on the second visual feature point, the second feature descriptor in each of the second image frames is matched with the feature descriptor of the first map to obtain a matching result, wherein the matching result is used to indicate the degree of matching between the second feature descriptor of the matched second image frame and the feature descriptor of the first map, wherein when the matching degree meets the preset matching requirement, it is determined that the current position of at least one of the above-mentioned shooting devices is successfully relocated.

[0063] In the embodiment of the present application, the 3D points and key frame information of the first map are pre-stored. Among them, the key frame information includes the feature points (i.e., 2D points) of that frame of image collected when the first map is established, and may also include the posture of the shooting device and the posture constraint relationship between the key frames, the constraint relationship between the key frames and the map points, etc. The selection of the key frame is related to the displacement and rotation of the shooting device. If the shooting device does not move, there is no need to add a new key frame. If the shooting device translates or rotates a certain threshold, it is necessary to add a new key frame, so that a stable and continuous shooting device posture and complete map information can be maintained.

[0064] Of course, the key frame information may also include the corresponding relationship between the 2D points and the 3D points, and the constraint relationship at this moment, and the constraint relationship includes the relationship between the edges of the pose graph optimization.

[0065] In an embodiment of the present application, for each second image frame, the second feature descriptor of the second visual feature point of the second image frame is matched with the feature descriptor of the first map. If the score obtained after matching is greater than a preset score threshold, the match is considered successful, that is, the repositioning of the shooting device is successful. Considering that the number of second image frames is greater than 1, and the number of successfully matched second image frames may also be greater than 1, therefore, only the second image frame corresponding to the highest score whose score obtained after matching is greater than the preset score threshold is judged as a successful match, and then the shooting device that shot the second image frame is relocated according to the matching result. Since the relative positions between the various shooting devices are fixed, as long as one shooting device is successfully relocated, the other shooting devices can also be successfully relocated.

[0066] In the embodiment of the present application, since the feature descriptor describes the feature points with useful information after discarding redundant information, the comparison speed can be increased by relocating the shooting device that took the second image frame by comparing the feature descriptor of the second image frame with the feature descriptor of the first map, thereby increasing the relocation speed.

[0067] In some embodiments, the step A3 of matching each second feature descriptor in each of the second image frames with the feature descriptors of the first map includes:

[0068] For any of the above second image frames, the following steps are performed:

[0069] A31. Determine the position information of the shooting device used to obtain the second image frame.

[0070] Specifically, the location information of the shooting device can be reflected by the location information of the device on which it is installed. For example, assuming that the shooting device is installed on a mobile device, the location information of the mobile device when the second image frame is obtained can be used as the location information of the shooting device. Of course, the location information of the shooting device can also be determined based on the installation position of the shooting device on the mobile device and the location information of the mobile device when the second image frame is obtained.

[0071] A32. Determine on the first map a map point that is within a specified distance from the location information.

[0072] After obtaining the location information of the shooting device, the 3D point cloud information corresponding to the location information is determined according to the 3D point cloud information of the first map, and then the 3D point cloud information adjacent to the 3D point cloud information is determined. For example, the map point corresponding to the 3D point cloud information within a specified distance from the 3D point cloud information is determined as the desired map point.

[0073] In some embodiments, KD-Tree can be used to store relevant information of the map points of the first map, such as storing the map points and the location information corresponding to the map points. After the location information of the shooting device is determined, the map points corresponding to the location information adjacent to the above location information can be searched through KD-Tree to improve the search speed.

[0074] A33. Project the determined map point onto the second image frame to obtain a two-dimensional projection point.

[0075] Since the map points are 3D (ie three-dimensional) point clouds, and the visual feature points of the second image frame are 2D (ie two-dimensional) feature points, in order to facilitate subsequent comparison, the map points need to be projected into the second image frame.

[0076] A34. Match the second feature descriptor of the second image frame with the feature descriptor of the two-dimensional projection point.

[0077] Specifically, a feature descriptor of a two-dimensional projection point is obtained, and then the feature descriptor of the two-dimensional projection point is compared with the second feature descriptor respectively. If the feature descriptor of a two-dimensional projection point is the same as a second feature descriptor, it indicates that the two-dimensional projection point is the same as the second feature point; otherwise, it indicates that the two-dimensional projection point is different from the second feature point.

[0078] In an embodiment of the present application, since the map points projected onto the second image frame are map points that are within a specified distance from the location information of the shooting device, after obtaining the two-dimensional projection points from these map points, the feature descriptor of the two-dimensional projection points is compared with the second feature descriptor, which can increase the rate of finding map points that match the second visual feature points (the visual feature points corresponding to the second feature descriptor), thereby increasing the rate of successful repositioning.

[0079] In some embodiments, after the above step A34, the following steps are included:

[0080] If the second feature descriptor of the second image frame does not match the feature descriptor of the two-dimensional projection point, a feature descriptor matching the second feature descriptor of the second image frame is searched for among all feature descriptors corresponding to the first map.

[0081] In the embodiment of the present application, when no feature descriptor matching the second feature descriptor is found during a local search in the first map, the search range is expanded. For example, the feature points of the first map and their corresponding feature descriptors are stored in a dictionary bag of words (DBoW) in advance, and after obtaining the feature points of the second image, the feature descriptor matching the second feature descriptor in the first map is searched through the DBoW. Since the search range is expanded, the success probability of relocalization can be increased.

[0082] In some embodiments, after searching for a feature descriptor matching the second feature descriptor of the second image frame among all feature descriptors corresponding to the first map, the method further includes:

[0083] B1. If no feature descriptor matching the second feature descriptor of the second image frame is found among all feature descriptors corresponding to the first map, a target body with a specified shape is added to the scene corresponding to the first map.

[0084] In the embodiment of the present application, after the repositioning of the photographing device in the first map fails, the matching rate can be increased by adding an easily identifiable target object to the scene.

[0085] The target body of the specified shape includes a rectangular QR code. For example, a QR code can be pasted in a scene, and after the QR code is added to the scene, the position of the QR code usually does not change. Since the corner points of the QR code are easy to detect, the mobile device can subsequently improve the success rate of constructing an incremental map based on the corner points of the QR code.

[0086] B2. Acquire at least two third image frames obtained by each of the above-mentioned shooting devices shooting the scene with the above-mentioned target object added.

[0087] In the embodiment of the present application, after the target object is added to the scene, image frames (i.e., third image frames) obtained by photographing the scene by various photographing devices are obtained, and therefore, image frames including the target object exist in the acquired third image frames.

[0088] B3. Extract visual feature points from each of the third image frames to obtain third visual feature points.

[0089] The third visual feature point may be an ORB feature point, and the feature descriptor corresponding to the third visual feature point is a BRISK descriptor.

[0090] B4. Determine corresponding map points according to each of the third visual feature points, and generate a third map according to the determined map points.

[0091] Specifically, the corresponding map points of each third visual feature point can be determined according to the binocular triangulation principle, and then the third map can be generated according to each map point.

[0092] In an embodiment of the present application, after the shooting device cannot be repositioned in the first map, the target body will be added to the scene corresponding to the first map. Since the target body has a specified shape, the third visual feature point is subsequently extracted from the image frame corresponding to the scene to which the target body is added, and after the third map is generated according to the third visual feature point, the third map will contain the information of the target body. In this way, when it is necessary to construct an incremental map based on the third map, the third map can be used as a new first map, and the incremental map can be constructed based on the new first map. Since the new first map contains the information of the target body, and the information of the target body is easy to detect, when the incremental map is subsequently constructed based on the new first map, the matching speed of the visual feature points can be improved, thereby improving the construction speed of the incremental map. For example, when the target body is a QR code, the four corner points of the QR code are first established in the first map, and each corner point has a unique ID, and the four corner points correspond to the ID of a QR code. In the process of QR code repositioning, if the four corner points of the QR code are detected in the current image frame, and the corresponding corner points are also searched in the first map, the repositioning is successful.

[0093] In some embodiments, if the matching degree between the second feature descriptor of the matched second image frame and the feature descriptor of the first map meets the preset matching requirement, the map construction method further includes:

[0094] C1. Determine the position and posture of a shooting device that shoots the matched second image frame according to the second visual feature points corresponding to the matched second image frame and the matched visual feature points of the first map.

[0095] Specifically, the position and posture of the camera can be determined according to the perspective-n-point (PnP) method. PnP is a method for solving the motion of 3D to 2D point pairs. It solves the problem of how to estimate the position and posture of the camera when we know n 3D space points and their projected positions in the 2D image.

[0096] C2. Update the visual inertial odometer of the mobile device according to the posture of the shooting device that shoots the matched second image frame, where the visual inertial odometer is used to estimate the posture of the mobile device.

[0097] Since the position and posture of the shooting device determined according to the image frame is more accurate, using the position and posture of the shooting device determined according to the image frame to update the Visual-Inertial Odometry (VIO) can improve the accuracy of the position and posture recorded by the VIO.

[0098] Correspondingly, the above step S15 specifically includes:

[0099] The position and posture of at least one of the above-mentioned photographing devices are determined according to the position and posture estimated by the above-mentioned visual inertial odometer.

[0100] In an embodiment of the present application, since determining the posture of the shooting device based on VIO is faster than determining the posture of the shooting device based on image frames, the posture of the shooting device can be determined based on the posture output by the VIO to increase the speed of generating the second map.

[0101] Of course, since VIO has cumulative errors, a time duration can be set. After the time duration is reached, the pose of the camera device is determined based on the new image frames captured by each camera device, and the pose of VIO is updated using the pose. Alternatively, after VIO cannot output a new pose, the pose of the camera device is determined based on the new image frames captured by each camera device, and the pose of VIO is updated using the pose.

[0102] In some embodiments, before the above step S16, it further includes:

[0103] Delete the outliers for the new map points.

[0104] Specifically, the outliers in each map point can be removed by using the Random Sample Consensus (RANSAC) method. Among them, RANSAC can robustly estimate model parameters. For example, it can estimate high-precision parameters from a data set containing a large number of outliers.

[0105] Correspondingly, the above step S16 includes:

[0106] The map points remaining after the external points are deleted are inserted into the first map to obtain the second map.

[0107] In the embodiment of the present application, since the outliers are outliers, the map points remaining after deleting the outliers are inserted into the first map, which can improve the accuracy of the obtained second map.

[0108] In some embodiments, the map construction method provided in the embodiments of the present application further includes:

[0109] D1. Perform loop detection on the image frames corresponding to the map points and the key frames of the first map to obtain loop detection results.

[0110] Specifically, the PnP method may be used to determine the 2D point corresponding to the map point, and then determine the image frame corresponding to the map point.

[0111] In some embodiments, the map point may be a map point remaining after the external point is deleted. If the map point is a map point that needs to be deleted, loop detection is not performed on the image frame corresponding to the map point.

[0112] D2. If the loop detection result indicates that the photographing device has passed through the position corresponding to the map point, globally optimize the first map.

[0113] Specifically, when the similarity between the current image frame and a key frame of the first map is greater than a preset similarity threshold, the loop detection result indicates that the shooting device has passed through the position corresponding to the map point corresponding to the key frame of the first map. After that, the first map will be globally optimized to obtain a second map with consistent global constraints.

[0114] In some embodiments, if the loop detection result indicates that the photographing device has never passed through the position corresponding to the map point, the first map is locally optimized.

[0115] In some embodiments, the first map may be optimized (e.g., global optimization or local optimization) using a bundle adjustment (BA) method. Since BA optimizes the state quantity with the minimum error by combining multiple frames, the state quantity is the position and map point of the shooting device (e.g., camera), adding the map point to the first map through BA optimization can improve the accuracy of the added map point.

[0116] In some embodiments, the first map may also be globally optimized based on a filtering method, which is not limited here.

[0117] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0118] Embodiment 2:

[0119] Corresponding to the map construction method of the above embodiment, Figure 2 A structural block diagram of a map construction device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0120] Reference Figure 2The map construction device 2 is applied to a mobile device including at least two shooting devices, the shooting ranges of the above-mentioned shooting devices have an intersection, and the relative positions between the above-mentioned shooting devices are fixed. The above-mentioned map construction device 2 includes: a repositioning module 21, a first image frame acquisition module 22, a first visual feature point determination module 23, a new map point determination module 24, a current posture determination module 25 of the shooting device, and a second map generation module 26. Among them:

[0121] The repositioning module 21 is used to reposition the current position of at least one of the above-mentioned shooting devices on a first map to obtain a repositioning result, wherein the above-mentioned first map is a constructed map.

[0122] The first image frame acquisition module 22 is used to acquire at least two first image frames after the above-mentioned repositioning result indicates that the current position of at least one of the above-mentioned shooting devices has been successfully repositioned, wherein each of the above-mentioned first image frames is respectively captured by different of the above-mentioned shooting devices at the same time.

[0123] The first visual feature point determination module 23 is used to extract visual feature points from each of the first image frames to obtain corresponding first visual feature points.

[0124] In some embodiments, the first visual feature point of the embodiment of the present application is an ORB feature point, and the feature descriptor corresponding to the first visual feature point is a BRISK descriptor.

[0125] The new map point determination module 24 is used to determine a new map point according to each of the above first visual feature points.

[0126] The current posture determination module 25 of the photographing device is used to determine the current posture of at least one of the above-mentioned photographing devices.

[0127] The second map generating module 26 is used to insert the new map point into the first map according to the new map point and the current position of at least one of the photographing devices to obtain a second map.

[0128] In the embodiment of the present application, since the first map is a constructed map, after relocating the shooting device on the first map, the new map point is determined according to the first image frame captured by each shooting device at the same time, which can ensure the accuracy of the coordinates, that is, the accuracy of the new map point can be guaranteed, and thus the accuracy of the generated second map can be improved. In addition, since the current position of the shooting device is also combined when inserting the new map point into the first map, and the current position of the shooting device can reflect the orientation of the shooting device, the position of the new map point can be quickly found in the first map in combination with the orientation of the shooting device, and then the new map point can be quickly inserted into the first map, thereby improving the speed of generating the second map. That is, in the embodiment of the present application, since the second map is generated on the basis of the first map, the generation speed of the second map is improved. At the same time, since the second map is a newly generated map, when the mobile device needs to be positioned, compared with positioning according to the first map, positioning according to the second map can improve the positioning success rate of the mobile device.

[0129] In some embodiments, the map construction device 2 further includes:

[0130] The positioning module is used to respond to the positioning instruction according to the second map when receiving the positioning instruction.

[0131] In some embodiments, the relocation module 21 includes:

[0132] The second image frame acquisition unit is used to acquire at least two second image frames, wherein each of the second image frames is captured by different of the above-mentioned shooting devices at the same time.

[0133] The second visual feature point determination unit is used to respectively determine the visual feature points and corresponding feature descriptors of each of the second image frames to obtain second visual feature points and second feature descriptors.

[0134] A feature matching unit is used to match the second feature descriptor in each of the second image frames with the feature descriptor of the first map based on the second visual feature point to obtain a matching result, wherein the matching result is used to indicate the degree of matching between the second feature descriptor of the matched second image frame and the feature descriptor of the first map, wherein when the matching degree meets the preset matching requirement, it is determined that the current position of at least one of the above-mentioned shooting devices is successfully relocated.

[0135] In some embodiments, when the feature matching unit matches each second feature descriptor in each of the second image frames with the feature descriptor of the first map, the method specifically includes:

[0136] For any of the above second image frames, the following steps are performed:

[0137] Determining position information of a photographing device used to obtain the second image frame;

[0138] Determining on the first map a map point within a specified distance from the location information;

[0139] Projecting the determined map point onto the second image frame to obtain a two-dimensional projection point;

[0140] The second feature descriptor of the second image frame is matched with the feature descriptor of the two-dimensional projection point.

[0141] In some embodiments, after matching the second feature descriptor of the second image frame with the feature descriptor of the two-dimensional projection point, the feature matching unit is further used to:

[0142] If the second feature descriptor of the second image frame does not match the feature descriptor of the two-dimensional projection point, a feature descriptor matching the second feature descriptor of the second image frame is searched for among all feature descriptors corresponding to the first map.

[0143] In some embodiments, the map construction device 2 further includes:

[0144] The target body adding module is used to add a target body with a specified shape to the scene corresponding to the first map if no feature descriptor matching the second feature descriptor of the second image frame is found among all feature descriptors corresponding to the first map.

[0145] The third image frame acquisition module is used to acquire at least two third image frames obtained by each of the above-mentioned shooting devices shooting the scene with the above-mentioned target object added.

[0146] The third visual feature point determination module is used to extract visual feature points from each of the third image frames to obtain third visual feature points.

[0147] The third map generation module is used to determine corresponding map points according to each of the third visual feature points, and generate a third map according to the determined map points.

[0148] In some embodiments, if the matching degree meets the preset matching requirement, the map construction device 2 further includes:

[0149] The posture determination module of the shooting device is used to determine the posture of the shooting device that shot the matched second image frame based on the second visual feature points corresponding to the matched second image frame and the visual feature points of the matched first map.

[0150] The visual inertial odometer posture update module is used to update the visual inertial odometer of the mobile device according to the posture of the shooting device that shoots the matched second image frame, and the visual inertial odometer is used to estimate the posture of the mobile device.

[0151] Correspondingly, the current posture determination module 25 of the shooting device is specifically used for:

[0152] The position and posture of at least one of the above-mentioned photographing devices are determined according to the position and posture estimated by the above-mentioned visual inertial odometer.

[0153] In some embodiments, the map construction device 2 further includes:

[0154] The outlier deletion module is used to delete the outlier points of the new map points.

[0155] The second map generation module 26 is specifically used for:

[0156] The map points remaining after the external points are deleted are inserted into the first map to obtain the second map.

[0157] In some embodiments, the map construction device 2 provided in the embodiment of the present application further includes:

[0158] The loop detection result determination module is used to perform loop detection on the image frame corresponding to the above map point and the key frame of the above first map to obtain the loop detection result.

[0159] Specifically, the PnP method may be used to determine the 2D point corresponding to the map point, and then determine the image frame corresponding to the map point.

[0160] In some embodiments, the map point may be a map point remaining after the external point is deleted. If the map point is a map point that needs to be deleted, loop detection is not performed on the image frame corresponding to the map point.

[0161] A global optimization module is used to perform global optimization on the first map if the loop detection result indicates that the shooting device has passed through the position corresponding to the map point.

[0162] Specifically, when the similarity between the current image frame and a key frame of the first map is greater than a preset similarity threshold, the loop detection result indicates that the shooting device has passed through the position corresponding to the map point corresponding to the key frame of the first map. After that, the first map will be globally optimized to obtain a second map with consistent global constraints.

[0163] In some embodiments, the global optimization module is further used for: if the loop detection result indicates that the photographing device has never passed through the position corresponding to the map point, then locally optimizing the first map.

[0164] In some embodiments, the first map may be optimized (e.g., global optimization or local optimization) using a bundle adjustment (BA) method. Since BA optimizes the state quantity with the minimum error by combining multiple frames, the state quantity is the position and map point of the shooting device (e.g., camera), adding the map point to the first map through BA optimization can improve the accuracy of the added map point.

[0165] In some embodiments, the first map may also be globally optimized based on a filtering method, which is not limited here.

[0166] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0167] Embodiment three:

[0168] Figure 3 This is a schematic diagram of the structure of a mobile device provided in an embodiment of the present application. Figure 3 As shown, the mobile device 3 of this embodiment includes: at least one processor 30 ( Figure 3 Only one processor is shown in the figure), a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, wherein the processor 30 implements the steps in any of the above-mentioned method embodiments when executing the computer program 32.

[0169] The mobile device 3 may be a robot, a navigation device, etc. The mobile device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of the mobile device 3 and does not constitute a limitation on the mobile device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0170] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0171] In some embodiments, the memory 31 may be an internal storage unit of the mobile device 3, such as a hard disk or memory of the mobile device 3. In other embodiments, the memory 31 may also be an external storage device of the mobile device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the mobile device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the mobile device 3. The memory 31 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0172] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0173] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.

[0174] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0175] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile device, the mobile device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / mobile device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0177] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0178] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0179] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0180] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A map construction method, characterized in that: Applied to a mobile device including at least two photographing devices, the photographing ranges of the photographing devices have an intersection, and the relative positions between the photographing devices are fixed, the map construction method includes: Relocating the current location of at least one of the photographing devices on a first map to obtain a relocation result, wherein the first map is a constructed map; After the relocation result indicates that the current position of at least one of the shooting devices has been successfully relocated, acquiring at least two first image frames, wherein each of the first image frames is captured by different shooting devices at the same time; Extracting visual feature points from each of the first image frames to obtain corresponding first visual feature points; Determine the position of the same position point in different first image frames according to each of the first visual feature points, then determine the coordinates of the position point in the world coordinate system according to each of the determined positions, and determine a new map point according to the coordinates; Determining a current position of at least one of the camera devices; According to the new map point and the current position of at least one of the photographing devices, the new map point is inserted into the first map to obtain a second map.

2. The map construction method according to claim 1, characterized in that: The step of relocating the current location of at least one of the photographing devices on the first map to obtain a relocation result includes: Acquire at least two second image frames, wherein each of the second image frames is captured by different capturing devices at the same time; Determine the visual feature points and the corresponding feature descriptors of each of the second image frames respectively, to obtain second visual feature points and second feature descriptors; Based on the second visual feature points, the second feature descriptor in each of the second image frames is matched with the feature descriptor of the first map to obtain a matching result, wherein the matching result is used to indicate the degree of matching between the second feature descriptor of the matched second image frame and the feature descriptor of the first map, wherein when the matching degree meets the preset matching requirements, it is determined that the current position of at least one of the shooting devices is successfully relocated.

3. The map construction method according to claim 2, characterized in that: The matching of each second feature descriptor in each second image frame with the feature descriptor of the first map comprises: For any one of the second image frames, the following steps are performed: Determining location information of a photographing device used to acquire the second image frame; Determining, on the first map, map points that are within a specified distance from the location information; Projecting the determined map point into the second image frame to obtain a two-dimensional projection point; The second feature descriptor of the second image frame is matched with the feature descriptor of the two-dimensional projection point.

4. The map construction method according to claim 3, characterized in that: After matching the second feature descriptor of the second image frame with the feature descriptor of the two-dimensional projection point, the method further comprises: If the second feature descriptor of the second image frame does not match the feature descriptor of the two-dimensional projection point, a feature descriptor matching the second feature descriptor of the second image frame is searched among all feature descriptors corresponding to the first map.

5. The map construction method according to claim 4, characterized in that: After searching for a feature descriptor matching the second feature descriptor of the second image frame among all feature descriptors corresponding to the first map, the method further includes: If no feature descriptor matching the second feature descriptor of the second image frame is found in all feature descriptors corresponding to the first map, adding a target body of a specified shape to the scene corresponding to the first map; Acquire at least two third image frames obtained by each of the shooting devices shooting the scene with the target object added thereto; Extracting visual feature points from each of the third image frames to obtain third visual feature points; Corresponding map points are determined according to each of the third visual feature points, and a third map is generated according to the determined map points.

6. The map construction method according to any one of claims 2 to 4, characterized in that: If the matching degree meets the preset matching requirement, the map construction method further includes: Determining, according to the second visual feature points corresponding to the matched second image frame and the matched visual feature points of the first map, the position and posture of the shooting device that shot the matched second image frame; updating a visual inertial odometer of the mobile device according to the posture of a shooting device that shoots the matched second image frame, wherein the visual inertial odometer is used to estimate the posture of the mobile device; Correspondingly, determining the current posture of at least one of the photographing devices includes: The pose of at least one of the photographing devices is determined according to the pose estimated by the visual inertial odometer.

7. The map construction method according to any one of claims 2 to 4, characterized in that: Before inserting the new map point into the first map according to the new map point and the current position of at least one of the photographing devices, the method includes: Deleting outliers from the new map points; The inserting the new map point into the first map according to the new map point and the current position of at least one of the photographing devices comprises: The map points remaining after the external points are deleted are inserted into the first map to obtain the second map.

8. A map construction device, characterized in that: Applied to a mobile device including at least two photographing devices, the photographing ranges of the photographing devices have an intersection, and the relative positions between the photographing devices are fixed, the map building device includes: a relocation module, configured to relocate the current location of at least one of the photographing devices on a first map to obtain a relocation result, wherein the first map is a constructed map; A first image frame acquisition module, configured to acquire at least two first image frames after the relocation result indicates that the current position of at least one of the shooting devices has been successfully relocated, wherein each of the first image frames is captured by different shooting devices at the same time; A first visual feature point determination module, used to extract visual feature points from each of the first image frames to obtain corresponding first visual feature points; a new map point determination module, used to determine the position of the same position point in different first image frames according to each of the first visual feature points, and then determine the coordinates of the position point in the world coordinate system according to each of the determined positions, and determine a new map point according to the coordinates; A current posture determination module of a photographing device, used to determine a current posture of at least one of the photographing devices; The second map generation module is used to insert the new map point into the first map according to the new map point and the current position of at least one of the photographing devices to obtain a second map.

9. A mobile device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images