Map data processing method and device, electronic device and mobile device
By acquiring and matching image features with the feature database of the target map in the robot, and combining visual and mapping sensor data, the problem of initial robot localization difficulty was solved, and efficient and accurate closed-loop detection and relocalization were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-16
- Publication Date
- 2026-03-27
AI Technical Summary
When the robot's initial localization is not set or the localization is lost, it is difficult to accurately find the corresponding position on the map, which makes closed-loop detection and relocalization difficult.
Accurate positioning is achieved by acquiring features from the image to be matched and matching them with a feature database collected from the same direction during the construction of the target map, combined with data from visual and mapping sensors.
This improves the accuracy and efficiency of robot location determination in the target map, and ensures the real-time performance and accuracy of closed-loop detection and relocation.
Smart Images
Figure CN117291974B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of map construction, and in particular to a map data processing method and device, an electronic device, and a mobile device. BACKGROUND
[0002] When a robot is constructing a map, loop closure detection is needed when returning to a place that has been passed before, a corresponding position in the map is found, and constraints are added for map optimization to eliminate the cumulative error of the map. When the robot is started for the first time without setting an initial position, or when the position is lost during task execution, an initial position needs to be given for repositioning. In the above two scenarios, how to find the association between the current position and the position on the map is the basis for loop closure detection and repositioning. SUMMARY
[0003] Therefore, embodiments of the present application provide a map data processing method and device, an electronic device, and a robot to solve the problem of matching the current position of the robot with the position on the map.
[0004] According to a first aspect, embodiments of the present application provide a map data processing method, comprising:
[0005] obtaining a to-be-matched image, the to-be-matched image being collected by an image collection device of a target mobile device at a current position along a target direction;
[0006] extracting features of the to-be-matched image to obtain to-be-matched features;
[0007] obtaining a feature database, the feature database being obtained by extracting features based on images collected along the target direction during target map construction, and the feature database being used to store features and corresponding position information;
[0008] determining a target feature corresponding to the to-be-matched features based on the similarity between the to-be-matched features and each feature in the feature database, to determine corresponding position information of the current position in the target map.
[0009] The map data processing method provided by the embodiments of the present application determines the position in the target map by collecting images, which can avoid the influence of the initial position on the matching result. Meanwhile, the images are collected along the same target direction, because there are differences between images collected in different directions at the same position. Therefore, whether the feature database is constructed or the image of the current position is collected, the images are collected based on the target direction. The consistency of the direction of the collected images is ensured by limiting the target direction, which improves the accuracy of determining the corresponding position of the current position in the target map.
[0010] In combination with the first aspect, in a first implementation of the first aspect, the obtaining the feature database comprises:
[0011] obtaining a collection direction of the image collection device in the process of constructing the target map, and calculating a deviation between the collection direction and the target direction;
[0012] adjusting the collection direction of the image collection device to the target direction based on the deviation;
[0013] obtaining image information collected by the image collection device in the target direction, the image information comprising an image;
[0014] extracting features of the image to construct the feature database.
[0015] The method for processing map data provided by the embodiment of the present application adjusts the collection direction of the image collection device to the target direction before image information is collected, and then collects the image and extracts the features, so that an accurate feature database is established, and the images in the feature database are all collected in the same target collection direction.
[0016] In combination with the first implementation of the first aspect, in a second implementation of the first aspect, the image information further comprises a second time stamp of collecting the image, and the extracting the features of the image to construct the feature database comprises:
[0017] extracting the features of the image;
[0018] obtaining a key frame corresponding to the target map and a first time stamp of the key frame;
[0019] determining a target key frame corresponding to the image based on a size relationship between the second time stamp and the first time stamp;
[0020] determining a pose corresponding to the image based on the pose of the target key frame;
[0021] binding the features of the image with the pose corresponding to the image to determine the feature database.
[0022] The method for processing map data provided by the embodiment of the present application finds the target key frame corresponding to the closest time stamp in the feature database by using the second time stamp of the collected image, and determines the pose corresponding to the image by using the target key frame, because the environment data of the position with close collection time stamps is not greatly different, and the key frame comprises the pose of the key frame, so that the binding relationship between the features and the pose is determined by using the target key frame corresponding to the time stamp, and thus the accurate feature database is constructed.
[0023] In a third implementation form of the first aspect, in conjunction with the second implementation form of the first aspect, the target key frame includes two key frames with the latest time stamps, and the determining the pose corresponding to the image based on the pose of the target key frame includes:
[0024] performing interpolation calculation based on the second time stamp, the first time stamp corresponding to the target key frame and the pose to determine the pose corresponding to the image.
[0025] The method for processing map data provided in the embodiments of the present application can obtain a richer feature database by using the second time stamp for interpolation calculation, so as to avoid the problem of inaccurate matching caused by too few features.
[0026] In a fourth implementation form of the first aspect, in conjunction with the first aspect, the determining the target feature corresponding to the to-be-matched feature based on the similarity between the to-be-matched feature and each feature in the feature database to determine the corresponding position information of the current position in the target map includes:
[0027] calculating the similarity between the to-be-matched feature and each feature;
[0028] determining the target feature with the highest similarity based on the calculation result of the similarity to obtain the pose corresponding to the target feature;
[0029] determining the corresponding position information of the current position in the target map by using the pose corresponding to the target feature.
[0030] The method for processing map data provided in the embodiments of the present application can improve the determination efficiency of the corresponding position by performing feature matching in the feature database, achieving preliminary positioning by using the result of feature matching first, narrowing the range, and then determining the corresponding position.
[0031] In a fifth implementation form of the first aspect, in conjunction with the fourth implementation form of the first aspect, the method further includes:
[0032] determining the pose corresponding to the target feature as a position initial value;
[0033] controlling the movement of the target mobile device to a target position corresponding to the position initial value;
[0034] obtaining acquisition data of a composition sensing device at the target position;
[0035] matching the acquisition data with the target map to determine a matching result.
[0036] The embodiment of the present application provides a map data processing method, similarity matching is performed by using visual features, a position closest to a current position is found, and then fine matching results are obtained by matching a mapping sensing device and a map. The method provides a coarse positioning result to simultaneous localization and mapping (SLAM), and then SLAM performs detailed positioning according to the coarse positioning result. The method is not limited to specific mapping sensing devices and algorithms.
[0037] According to a second aspect, the embodiment of the present application further provides a map data processing device, comprising:
[0038] A first obtaining module is configured to obtain a to-be-matched image, wherein the to-be-matched image is an image collected by an image collection device of a target mobile device at a current position along a target direction;
[0039] An extracting module is configured to extract features of the to-be-matched image to obtain to-be-matched features;
[0040] A second obtaining module is configured to obtain a feature database, wherein the feature database is obtained by extracting features based on images collected along the target direction when a target map is constructed, and the feature database is configured to store features and corresponding position information;
[0041] A determining module is configured to determine target features corresponding to the to-be-matched features based on similarities between the to-be-matched features and each feature in the feature database, so as to determine corresponding position information of the current position in the target map.
[0042] According to a third aspect, the embodiment of the present application provides an electronic device, comprising a memory and a processor, which are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions, thereby executing the map data processing method in the first aspect or any one of the embodiments of the first aspect.
[0043] According to a fourth aspect, the embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make the computer execute the map data processing method in the first aspect or any one of the embodiments of the first aspect.
[0044] According to a fifth aspect, the embodiment of the present application further provides a mobile device, comprising:
[0045] An image collection device is configured to collect images.
[0046] The electronic device in the third aspect of the present application is connected with the image collection device.
[0047] The mobile device provided by the embodiment of the present application can obtain an accurate and perfect feature database during the composition process by setting the image acquisition device and the electronic device, so that the corresponding result can be efficiently obtained when repositioning or closed loop detection is needed, and the real-time performance of map data processing is improved. BRIEF DESCRIPTION OF DRAWINGS
[0048] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0049] Figure 1 FIG. 1 is a structural schematic diagram of a mobile device according to an embodiment of the present application;
[0050] Figure 2 FIG. 2 is a hardware structural schematic diagram of an electronic device according to an embodiment of the present application;
[0051] Figure 3 FIG. 3 is a flowchart of a map data processing method according to an embodiment of the present application;
[0052] Figure 4 FIG. 4 is a flowchart of a map data processing method according to an embodiment of the present application;
[0053] Figure 5 FIG. 5 is a flowchart of a map data processing method according to an embodiment of the present application;
[0054] Figure 6 FIG. 6 is a structural block diagram of a map data processing device according to an embodiment of the present application. DETAILED DESCRIPTION
[0055] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0056] For the application of visual feature-based methods in associating the current location with a location on a map, during map construction, environmental images are acquired and their scene features are extracted, then stored in a one-to-one correspondence with the acquired locations. Whenever loop closure detection or relocalization is required, visual features are extracted from the image acquired at the current location and matched with existing visual features. After finding the most similar visual feature, and based on its corresponding location T, mapping sensors, such as vision or LiDAR, are used to perform a fine-grained match between location T and the map, using location T as the predicted value.
[0057] The aforementioned visual feature-based method uses image feature similarity to find a similar location as an initial value, and then performs loop closure detection and relocalization through fine matching. However, commonly used cameras typically have a very small field of view, only acquiring partial information about the environment. The robot can only acquire similar images and obtain high feature similarity when it returns to the same location with a similar camera angle.
[0058] Based on this, this invention proposes a map data processing method with a fixed image acquisition direction. Before each image acquisition, the acquisition direction of the image acquisition device is adjusted to align with the target direction. Following this, image acquisition is performed and features are extracted. The extracted features are then matched with features in a feature database to determine the most similar target features. Subsequently, based on this, the acquisition results from the mapping sensor are combined for fine-tuning, achieving accurate relocation or loop closure detection.
[0059] The map data processing method provided in this embodiment of the invention combines visual features with data collected by mapping sensors to achieve real-time and accurate relocation or loop closure detection.
[0060] This invention provides a mobile device, including but not limited to mobile robots, self-moving devices, etc. For example... Figure 1 As shown, the mobile device includes an electronic device 10 and an image acquisition device 20. The image acquisition device 20 is connected to the electronic device 10 and sends the acquired image to the electronic device 10. The electronic device 10 determines the corresponding position of the current location of the acquired image on the target map by executing the map data processing method of this embodiment. Subsequently, based on this, relocation or loop closure detection is performed as needed. By setting up the image acquisition device and the electronic device, an accurate and complete feature database can be obtained simultaneously during the mapping process, which facilitates efficient acquisition of corresponding results when relocation or loop closure detection is required, thus improving the real-time performance of map data processing.
[0061] The mobile device also includes a mapping sensor for real-time data acquisition and simultaneous localization and mapping (SLAM) to build a target map. During SLAM, the image acquisition device captures images along the target direction, constructing a feature database. The mapping sensor includes, but is not limited to, 2D / 3D LiDAR or RGB vision.
[0062] In some optional embodiments, the mobile device further includes a directional device connected to the electronic device for acquiring the acquisition direction of the image acquisition device. The directional device sends the acquired acquisition direction to the electronic device, which adjusts the acquisition direction of the image acquisition device to the target direction based on the difference between the acquired direction and the target direction. The directional device may include, but is not limited to, a gyroscope, a magnetometer, etc., and is configured according to actual needs.
[0063] This invention also provides an electronic device having the following features. Figure 6 The device shown is for processing map data.
[0064] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 2 As shown, the electronic device may include: at least one processor 101, such as a CPU (Central Processing Unit), at least one communication interface 103, memory 104, and at least one communication bus 102. The communication bus 102 is used to enable communication between these components. The communication interface 103 may include a display screen or a keyboard; optionally, the communication interface 103 may also include a standard wired interface or a wireless interface. The memory 104 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 104 may also be at least one storage device located remotely from the aforementioned processor 101. The processor 101 may be combined with... Figure 6 The described apparatus has an application program stored in memory 104, and the processor 101 calls the program code stored in memory 104 to perform any of the following method steps.
[0065] The communication bus 102 can be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The communication bus 102 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 2 Only one thick line is used to represent the communication bus 102, but it does not mean that there is only one bus or only one type of bus.
[0066] The memory 104 can include a volatile memory, such as a random-access memory (RAM), and can also include a non-volatile memory, such as a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). The memory 104 can also include a combination of the above-mentioned types of memories.
[0067] The processor 101 can be a central processing unit (CPU), a network processor (NP), or a combination of the CPU and the NP.
[0068] The processor 101 can further include a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0069] Optionally, the memory 104 is further configured to store program instructions. The processor 101 can invoke the program instructions to implement the method for processing map data as shown in any embodiment of the present application.
[0070] The electronic device in the embodiment of the present application can be integrated in the mobile device or set in the background server. When the electronic device is set in the background server, the mobile device comprises an image acquisition device, which sends the acquired image to the background server. Correspondingly, the electronic device in the background server determines the corresponding position of the image acquisition device in the target map when the image acquisition device acquires the image by executing the processing method of map data provided in the embodiment of the present application, so as to control the mobile device to perform repositioning or closed loop detection and the like.
[0071] As an application scenario of the processing method of map data provided in the embodiment of the present application, the mobile device integrates the electronic device, the image acquisition device and the orientation device. In the SLAM process of the mobile device, the orientation device adjusts the acquisition direction of the image acquisition device to the target direction by acquiring the acquisition direction of the image acquisition device, and the image acquisition device acquires the image along the target direction to construct the feature database. When repositioning or closed loop detection is needed, the image acquisition device acquires the image along the target direction, and determines the corresponding position of the current position on the target map in combination with the feature database. The corresponding position is taken as the initial position, and then the positioning is accurately performed in combination with the composition sensor, that is, the repositioning or closed loop detection is completed.
[0072] As an application scenario of the processing method of map data provided in the embodiment of the present application, the electronic device is integrated in the background server. The image acquisition device in the mobile device acquires the image along the target direction and sends the image to the electronic device in the background server. The electronic device executes the processing method of map data provided in the embodiment of the present application and communicates with the mobile device, and performs repositioning or closed loop detection on the mobile device in combination with the composition sensor arranged on the mobile device.
[0073] According to the embodiment of the present application, a processing method of map data is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in different order from here.
[0074] In the embodiment, a processing method of map data is provided, which can be used for the electronic device, Figure 3 is a flowchart of the processing method of map data according to the embodiment of the present application, as Figure 3 shown, the flowchart comprises the following steps:
[0075] S11, acquiring a to-be-matched image.
[0076] The image to be matched is captured by the image acquisition device of the target mobile device at the current position along the target direction.
[0077] In some application scenarios, it is necessary to determine the target mobile device's current location on a target map while it is moving. Based on this, at the target mobile device's current location, an image acquisition device acquires images to be matched along the target direction. The target direction is a preset, fixed direction.
[0078] The image acquisition device is mounted on the target mobile device. The image acquisition device can be fixed and non-rotatable, or it can be rotatable. Specifically, when the image acquisition device is fixed and non-rotatable, the rotation of the target mobile device is controlled to adjust the acquisition direction, thereby causing the image acquisition device to rotate. When the image acquisition device is rotatable, its rotation is controlled to adjust the acquisition direction to the target direction.
[0079] In some implementations, the image acquisition device is a gimbal camera or a fixed camera, and the orientation device is a magnetometer. That is, the target mobile device includes a magnetometer and a gimbal camera or a fixed camera. If a gimbal camera is installed on the target mobile device, during the mapping, loop closure detection, or repositioning process, the gimbal needs to be continuously controlled to acquire images in the target direction indicated by the magnetometer. If a fixed camera is used on the target mobile device, during the mapping stage, at the key position generated, the device needs to be stopped and rotated to the corresponding angle direction of the magnetometer to take a picture. Similarly, the same operation is performed during loop closure detection or repositioning.
[0080] S12, extract the features of the image to be matched to obtain the matching features.
[0081] After acquiring the image to be matched, the electronic device extracts its features. For example, it may use a feature extraction model to extract features and obtain the features to be matched; or it may use image analysis methods to extract features from the image to be matched, and so on. There are no specific limitations on the method of extracting features from the image to be matched; the method is set according to actual needs.
[0082] S13, Obtain the feature database.
[0083] The feature database is obtained by extracting features from images collected along the target direction during the construction of the target map, and the feature database is used to store features and their corresponding location information.
[0084] As described above, during the mapping process, the image acquisition device acquires images along the target direction and sends the images to the electronic device, and the electronic device extracts features from the images to construct the feature database. During the image acquisition, the electronic device records the position information of the target mobile device in the target map, and therefore, the electronic device can establish the correspondence between the features and the position information to form the feature database.
[0085] The feature database can be stored in the electronic device in advance or obtained from a third-party device, and the source of the feature database is not limited.
[0086] The step will be described in detail below.
[0087] S14, determining the target feature corresponding to the to-be-matched feature based on the similarity between the to-be-matched feature and each feature in the feature database, to determine the corresponding position information of the current position in the target map.
[0088] The electronic device calculates the similarity between the to-be-matched feature and each feature in the feature database based on the to-be-matched feature obtained in S12, and determines the feature with the highest similarity in the feature database as the target feature. As described above, the feature database is used to store features and corresponding position information, and therefore, after the target feature is determined, the position information corresponding to the target feature can be obtained, and the corresponding position information of the current position in the target map can be determined.
[0089] The step will be described in detail below.
[0090] The mapping data processing method provided in this embodiment determines the position in the target map by acquiring images, which can avoid the influence of the initial position on the matching result. In addition, the images are acquired along the same target direction, and there is a difference between the images acquired in different directions at the same position. Therefore, the construction of the feature database and the image acquisition of the current position are both based on the images acquired along the target direction, and the consistency of the acquired images is ensured by limiting the target direction, which improves the accuracy of determining the corresponding position of the current position in the target map.
[0091] In this embodiment, a mapping data processing method is provided, which can be used in the electronic device described above, Figure 4 is a flowchart of the mapping data processing method according to an embodiment of the present application, as shown in the figure, the flowchart includes the following steps: Figure 4
[0092] S21, acquiring a to-be-matched image.
[0093] The to-be-matched image is an image collected by an image collection device of the target mobile device at a current position along a target direction.
[0094] For details, please refer to Figure 3 S11 of the illustrated embodiment, which will not be repeated here.
[0095] S22, extracting features of the to-be-matched image to obtain to-be-matched features.
[0096] For details, please refer to Figure 3 S12 of the illustrated embodiment, which will not be repeated here.
[0097] S23, obtaining a feature database.
[0098] The feature database is obtained by feature extraction based on images collected along the target direction when the target map is constructed, and is used to store features and corresponding position information.
[0099] Specifically, the above S23 includes:
[0100] S231, obtaining a collection direction of the image collection device in the process of constructing the target map, and calculating a deviation of the collection direction from the target direction.
[0101] The construction of the target map and the construction of the feature database are performed simultaneously, that is, the mapping sensor collects sensing data to construct the target map, and in this process, the image collection device collects images to construct the feature database. Before image collection is performed by the image collection device, it is necessary to determine whether the collection direction of the image collection device is consistent with the target direction, and if not, the collection direction of the image collection device needs to be adjusted.
[0102] When a directional device is provided in the target mobile device, the collection direction of the image collection device is determined by the directional device, and correspondingly, the electronic device calculates the deviation of the collection direction from the target direction. For example, the directional device is a magnetometer, and the electronic device reads the magnetometer angle data a and calculates the difference between it and the target angle G.
[0103] S232, adjusting the collection direction of the image collection device to the target direction based on the deviation.
[0104] As described above, if the image collection device is not rotatable, the electronic device controls the rotation of the target mobile device based on the deviation to adjust the collection direction of the image collection device to the target direction; if the image collection device is a camera with a gimbal, the electronic device controls the rotation of the gimbal based on the deviation to drive the rotation of the image collection device, thereby adjusting its collection direction to the target direction. Continuing with the above example, after the deviation of the magnetometer angle data from the target angle is calculated, the collection direction of the image collection device is adjusted to the target direction based on the deviation.
[0105] S233, acquire image information collected by the image collection device in the target direction.
[0106] The image information includes an image.
[0107] After the collection direction of the image collection device is adjusted to the target direction, the image collection device is triggered to collect image information to construct an image database.
[0108] S234, extract features of the image to construct a feature database.
[0109] The electronic device extracts features from the image collected in S233, and the feature database includes features and corresponding position information. Based on this, the electronic device establishes a correspondence between features and position information to obtain the feature database.
[0110] In some optional embodiments, the image information further includes a second timestamp of the collected image. When collecting the image using the image collection device, the image collection device also records the time of collecting the image, i.e., the second timestamp. Based on this, S234 described above includes:
[0111] (1) extract features of the image.
[0112] (2) acquire a key frame corresponding to the target map and a first timestamp of the key frame.
[0113] The electronic device controls the movement of the target mobile device to perform SLAM mapping to obtain a series of key frames and corresponding timestamps {(K1, KT1), (K2, KT2), (K3, KT3),..., (K n , KT n )}. K j is the jth key frame, KT j is the first timestamp corresponding to the jth key frame. Each key frame K j contains the pose Pose j of the key frame.
[0114] (3) determine the target key frame corresponding to the image based on the size relationship between the second timestamp and the first timestamp.
[0115] The electronic device compares the second timestamp with each first timestamp to determine the first timestamp closest in time to the second timestamp, and determines the key frame corresponding to the closest first timestamp as the target key frame corresponding to the image. The number of closest first timestamps is set according to actual requirements, for example, 1, 2, or more.
[0116] (4) determine the pose corresponding to the image based on the pose of the target key frame.
[0117] After determining the target key frame, the electronic device extracts the pose of the target key frame and determines the pose corresponding to the image.
[0118] In some embodiments, the target key frame includes two key frames with the most recent timestamps, i.e., the two key frames corresponding to the closest first timestamps are extracted. The step (4) includes: determining the pose corresponding to the image based on the second timestamp, the first timestamp corresponding to the target key frame, and the pose.
[0119] For example, the electronic device finds two target key frames (K j , KT j ) and (K j+1 , KT j+1 ) with the most recent timestamps in the SLAM map using the second timestamp. And calculates the pose corresponding to the image according to the time interpolation:
[0120] F_pose i = Pose j *(Pose j+1 *Pose j -1 *(T i -KT j ) / (KT j+1 -KT j ))
[0121] The interpolation calculation using the second timestamp can obtain a richer feature database to avoid the problem of inaccurate matching caused by fewer features.
[0122] (5) Bind the features of the image with the pose corresponding to the image to determine the feature database.
[0123] The electronic device binds the features F i of the image P i with the corresponding pose F_pose i to obtain {(F1, F_pose1), (F2, F_pose2), (F3, F_pose3),..., (F n , F_pose n )} and stores it, i.e., obtains the feature database.
[0124] The second timestamp of the collected image is used to find the target key frame corresponding to the closest timestamp in the feature database, and the pose corresponding to the image is determined by using the target key frame, because the environment data of the position with a close collection timestamp is not much different, the key frame includes the pose of the key frame, so that the binding relationship between the feature and the pose can be obtained by determining the corresponding target key frame by using the timestamp, so as to construct an accurate feature database.
[0125] S24, determining the target feature corresponding to the to-be-matched feature based on the similarity of the to-be-matched feature and each feature in the feature database, to determine the corresponding position information of the current position in the target map.
[0126] For details, please refer to Figure 3 S14 of the embodiment shown, which will not be repeated here.
[0127] The map data processing method provided in the embodiment adjusts the collection direction of the image collection device to a target direction before image information collection, and then performs image collection and feature extraction, so as to establish an accurate feature database, so that the images in the feature database are all collected in the same target collection direction.
[0128] As a specific application example of the embodiment, the image collection device is a camera with a gimbal, and the orientation device is a magnetometer. Based on this, the method comprises:
[0129] 1. Target map construction, calculating the collection direction by using the magnetometer, and controlling the gimbal to rotate and take pictures
[0130] (1) Set the target angle G, which can be set to an arbitrary fixed angle value;
[0131] (2) The electronic device controls the movement of the target mobile device to perform SLAM mapping, and obtains a series of key frames and corresponding times {(K1, KT1), (K2, KT2), (K3, KT3),..., (K n , KT n )}; K j is the jth key frame, and KT j is the first timestamp corresponding to the jth key frame. Each key frame K j contains the pose Pose j of the key frame.
[0132] (3) Read the angle data a of the magnetometer, calculate the difference with the target angle G, control the gimbal to rotate to the target angle G, take pictures, and obtain the corresponding image P i and the picture taking time T i ;
[0133] (4) the target mobile device repeats (3) during the mapping process until the mapping ends, obtaining a series of images and second timestamps {(P1, T1), (P2, T2), (P3, T3),..., (P m , T m )};
[0134] 2. Extract image features and establish a feature database
[0135] (1) The electronic device uses a deep neural network model such as NetVALD to extract image features Fi from the image Pi, Fi being a one-dimensional vector, obtaining {(F1, T1), (F2, T2), (F3, T3),..., (F m , T m )};
[0136] (2) According to the second timestamp, obtain the two key frames (K j , KT j ) and (K j+1 , KT j+1 ) closest in time to the SLAM map;
[0137] (3) Calculate the feature F i of the image P i corresponding to the pose F_pose i according to time interpolation;
[0138] (4) Bind the feature F i of the image P i to the corresponding pose F_pose i , obtaining {(F1, F_pose1), (F2, F_pose2), (F3, F_pose3),..., (F n , F_pose n )} and storing it, that is, obtaining the feature database.
[0139] Since visual-based loop closure detection or relocalization is greatly limited by the angle of view, the image features obtained from images taken at different angles at the same location often differ greatly. Based on this, the embodiment controls the photographing direction based on the magnetometer, ensures that the target mobile device always photographs in one direction, ensures the same angle of view, and improves the success rate of relocalization or loop closure detection.
[0140] In this embodiment, a map data processing method is provided, which can be used in the above-mentioned electronic device, Figure 5 is a flowchart of the map data processing method according to an embodiment of the present application, as shown in Figure 5 , the flowchart includes the following steps:
[0141] S31, obtaining an image to be matched.
[0142] The to-be-matched image is an image collected by an image collection device of the target mobile device at a current position along a target direction.
[0143] For details, please refer to Figure 3 S11 of the illustrated embodiment, which will not be repeated here.
[0144] S32, extracting features of the to-be-matched image to obtain to-be-matched features.
[0145] For details, please refer to Figure 3 S12 of the illustrated embodiment, which will not be repeated here.
[0146] S33, obtaining a feature database.
[0147] The feature database is obtained by feature extraction based on images collected along a target direction when a target map is constructed, and is used to store features and corresponding position information.
[0148] For details, please refer to Figure 4 S23 of the illustrated embodiment, which will not be repeated here.
[0149] S34, determining a target feature corresponding to the to-be-matched feature based on the similarity between the to-be-matched feature and each feature in the feature database, to determine corresponding position information of the current position in the target map.
[0150] Specifically, the above S34 includes:
[0151] S341, calculating the similarity between the to-be-matched feature and each feature.
[0152] The electronic device can calculate the similarity between the to-be-matched feature and each feature in the feature database by using a cosine distance or other calculation method. The calculation method of the similarity is not limited here and can be set according to actual needs.
[0153] S342, determining a target feature with the highest similarity based on the calculation result of the similarity, to obtain a pose corresponding to the target feature.
[0154] The electronic device extracts the feature with the highest similarity from the calculated similarity and takes it as the target feature. As described above, the feature database stores features and corresponding position information, and the position information is a pose in this embodiment. Based on this, after the target feature is determined, the electronic device extracts the pose corresponding to the target feature from the feature database.
[0155] S343, determining corresponding position information of the current position in the target map by using the pose corresponding to the target feature.
[0156] S35, determine the pose corresponding to the target feature as the position initial value.
[0157] S36, control the target mobile device to move to the target position corresponding to the position initial value.
[0158] The electronic device sets the pose corresponding to the target feature as the position initial value, and controls the target mobile device to move to the target position corresponding to the position initial value, which is used as the starting point of the map search. By finding the pose most similar to the image of the current position, the search range is narrowed, that is, the search matching is performed near the most similar position.
[0159] S37, acquire the acquisition data of the composition sensing device at the target position.
[0160] S38, match the acquisition data with the target map to determine the matching result.
[0161] The determined pose F_pose is used as the position initial value, and the acquisition data of the composition sensor is matched with the target map to obtain a fine positioning result, that is, the repositioning or loop detection is completed. For example, when the composition sensor is a 2D / 3D laser, the ICP, NDT and other point cloud matching methods are used to match with the target map; when the composition sensor is a visual sensor, the visual feature point matching method is used to match with the target map.
[0162] Vision is the most abundant information that robots can obtain, and has a high success rate for repositioning and loop detection. In this embodiment, the vision-based repositioning or loop detection is applied to different SLAM algorithms, such as 2D / 3D laser SLAM or RGB image-based SLAM algorithm.
[0163] The map data processing method provided in this embodiment performs feature matching in the feature database, first uses the feature matching result to realize preliminary positioning, narrows the range, and then determines the corresponding position, which can improve the determination efficiency of the corresponding position. The similarity matching is performed using the visual feature to find the position closest to the current position, and then the composition sensing device is matched with the map to obtain a fine matching result. This method first provides a coarse positioning result to the SLAM, and then the SLAM performs detailed positioning according to the coarse positioning result. This method is not limited to the specific composition sensing device and algorithm used.
[0164] A map data processing apparatus is also provided in the embodiments, which is configured to implement the embodiments and preferred embodiments described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
[0165] A map data processing apparatus is provided in the embodiments, which is configured to implement the embodiments and preferred embodiments described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated. Figure 6
[0166] A first obtaining module 41 is configured to obtain a to-be-matched image, which is collected by an image collection device of a target mobile device at a current position along a target direction;
[0167] A feature extraction module 42 is configured to extract features of the to-be-matched image to obtain to-be-matched features;
[0168] A second obtaining module 43 is configured to obtain a feature database, which is obtained by feature extraction based on images collected along the target direction when a target map is constructed, and is configured to store features and corresponding position information;
[0169] A determining module 44 is configured to determine a target feature corresponding to the to-be-matched features based on similarity between the to-be-matched features and each feature in the feature database, so as to determine corresponding position information of the current position in the target map.
[0170] In some embodiments, the second obtaining module 43 comprises:
[0171] A first obtaining unit is configured to obtain a collection direction of the image collection device during construction of the target map, and calculate a deviation between the collection direction and the target direction;
[0172] An adjusting unit is configured to adjust the collection direction of the image collection device to the target direction based on the deviation;
[0173] A second obtaining unit is configured to obtain image information collected by the image collection device along the target direction, wherein the image information comprises images;
[0174] A feature extraction unit is configured to extract features of the images to construct the feature database.
[0175] In some embodiments, the image information further comprises a second timestamp of collecting the images, and the feature extraction unit comprises:
[0176] A feature extraction subunit is configured to extract features of the images;
[0177] The acquisition subunit is configured to acquire a key frame corresponding to the target map and a first timestamp of the key frame.
[0178] The first determination subunit is configured to determine a target key frame corresponding to the image based on a size relationship between the second timestamp and the first timestamp.
[0179] The second determination subunit is configured to determine a pose corresponding to the image based on a pose of the target key frame.
[0180] The binding subunit is configured to bind a feature of the image and the pose corresponding to the image, and determine the feature database.
[0181] In some embodiments, the target key frame includes two key frames with the most recent timestamps, and the second determination subunit includes:
[0182] The third determination subunit is configured to determine the pose corresponding to the image based on interpolation calculation of the second timestamp, a first timestamp corresponding to the target key frame, and a pose.
[0183] In some embodiments, the determination module 44 includes:
[0184] The calculation unit is configured to calculate a similarity between the to-be-matched feature and each feature.
[0185] The first determination unit is configured to determine a target feature with the highest similarity based on a calculation result of the similarity, so as to obtain a pose corresponding to the target feature.
[0186] The second determination unit is configured to determine corresponding position information of the current position in the target map by using the pose corresponding to the target feature.
[0187] In some embodiments, the map data processing apparatus further includes:
[0188] The pose determination module is configured to determine the pose corresponding to the target feature as a position initial value.
[0189] The control module is configured to control movement of the target mobile device to a target position corresponding to the position initial value.
[0190] The third acquisition module is configured to acquire acquisition data of a composition sensing device at the target position.
[0191] The matching module is configured to match the acquisition data with the target map, and determine a matching result.
[0192] The processing device of the map data in the embodiment is presented in the form of functional units, and the units herein refer to ASIC circuits, processors and memories for executing one or more software or fixed programs, and / or other devices that can provide the above functions.
[0193] Further function descriptions of the above modules are the same as the corresponding embodiments described above, and will not be described here.
[0194] The embodiment of the present application further provides a non-transitory computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the map data processing method in any method embodiment described above. The storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.
[0195] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.
Claims
1. A method of processing map data, characterized by, The method comprises: obtaining a to-be-matched image, the to-be-matched image being an image collected by an image collection device of a target mobile device at a current position along a target direction; extracting features of the to-be-matched image to obtain to-be-matched features; obtaining a feature database, the feature database being obtained by feature extraction based on images collected along the target direction when a target map is constructed, and the feature database being used to store features and corresponding position information; determining a target feature corresponding to the to-be-matched feature based on a similarity between the to-be-matched feature and each feature in the feature database, to determine corresponding position information of the current position in the target map; wherein the obtaining of the feature database comprises: obtaining a collection direction of the image collection device in the process of constructing the target map, and calculating a deviation between the collection direction and the target direction; adjusting the collection direction of the image collection device to the target direction based on the deviation; obtaining image information collected by the image collection device along the target direction, the image information comprising an image and a second time stamp at which the image is collected; extracting features of the image; obtaining corresponding key frames of the target map and first time stamps of the key frames; determining a target key frame corresponding to the image based on a size relationship between the second time stamp and the first time stamp, the target key frame comprising two key frames with the most recent time stamps; performing interpolation calculation based on the second time stamp, the first time stamp corresponding to the target key frame, and a pose, to determine a pose corresponding to the image; binding the features of the image with the pose corresponding to the image, to determine the feature database.
2. The method of claim 1, wherein, The determining of the target feature corresponding to the to-be-matched feature based on the similarity between the to-be-matched feature and each feature in the feature database, to determine corresponding position information of the current position in the target map, comprises: calculating the similarity between the to-be-matched feature and each feature; determining a target feature with the highest similarity based on a calculation result of the similarity, to obtain a pose corresponding to the target feature; determining corresponding position information of the current position in the target map by using the pose corresponding to the target feature.
3. The method of claim 2, wherein, The method further comprises: determining a pose corresponding to the target feature as an initial position value; controlling movement of the target mobile device to a target position corresponding to the initial position value; obtaining collection data of a mapping sensor device at the target position; matching the collection data with the target map to determine a matching result.
4. A map data processing device, characterized by comprising: The method comprises: a first obtaining module, configured to obtain a to-be-matched image, the to-be-matched image being an image collected by an image collection device of a target mobile device at a current position along a target direction; an extracting module, configured to extract features of the to-be-matched image to obtain to-be-matched features; a second obtaining module, configured to obtain a feature database, the feature database being obtained by feature extraction based on images collected along the target direction when a target map is constructed, and the feature database being used to store features and corresponding position information; The determining module is configured to determine the target feature corresponding to the to-be-matched feature based on a similarity between the to-be-matched feature and each feature in the feature database, so as to determine the corresponding position information of the current position in the target map. The feature database is obtained by: During the construction of the target map, the acquisition direction of the image acquisition device is obtained, and a deviation between the acquisition direction and the target direction is calculated; The acquisition direction of the image acquisition device is adjusted to the target direction based on the deviation; Image information acquired by the image acquisition device in the target direction is obtained, the image information including an image and a second timestamp of acquiring the image; Features of the image are extracted; A key frame corresponding to the target map and a first timestamp of the key frame are obtained; Based on a size relationship between the second timestamp and the first timestamp, a target key frame corresponding to the image is determined, the target key frame including two key frames with the most recent timestamps; Based on the second timestamp, the first timestamp corresponding to the target key frame, and a pose, an interpolation calculation is performed to determine a pose corresponding to the image; The features of the image are bound to the pose corresponding to the image, and the feature database is determined.
5. An electronic device, comprising: The electronic device comprises: a memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the map data processing method of any one of claims 1-3.
6. A computer readable storage medium characterized by The computer readable storage medium stores computer instructions for causing a computer to perform the map data processing method of any one of claims 1-3.
7. A mobile device, characterized by The electronic device comprises: an image acquisition device configured to acquire images; The electronic device of claim 5 is connected to the image acquisition device.
Citation Information
Patent Citations
Indoor positioning method and device of intelligent mobile terminal and electronic equipment
CN111199564A