A method, apparatus, electronic device, and storage medium for generating mapping data

Abstract

An embodiment of the present application provides a method, apparatus, electronic device, and storage medium for generating mapping data, which relates to the technical field of robot mapping. The method includes: obtaining each to-be-utilized image collected by an image acquisition device and the corresponding to-be-utilized pose when the robot travels in a to-be-mapped scene; calculating the three-dimensional coordinates of the real positions represented by each effective feature point; calculating the relative loop observation pose between the to-be-utilized images with a loop relationship; using the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image, the association relationship between each effective feature point and the represented real position, and the relative loop observation pose between the to-be-utilized images with a loop relationship as constraint conditions for data optimization, and obtaining the final true pose of the robot when each to-be-utilized image is collected and the final true coordinates of the real positions represented by each effective feature point as the current mapping data.

Description

Technical Field

[0001] The present application relates to the technical field of robot mapping, and particularly to a method, device, electronic device, and storage medium for generating mapping data. Background Art

[0002] The SLAM (Simultaneous Localization and Mapping) technology is a technology that enables a robot to perform autonomous localization and navigation in an unknown environment. In order to enable the robot to accurately locate, it is first necessary to generate accurate mapping data.

[0003] However, in order to generate accurate mapping data, it is necessary for the robot to be equipped with a variety of acquisition devices to collect information of the scene to be mapped. For example, it is necessary to install a stereo camera, an inertial measurement unit, a lidar, etc. in the robot, and the cost of these acquisition devices is relatively high, resulting in a relatively high cost of the robot for generating mapping data.

[0004] Therefore, how to reduce the cost of the robot while generating accurate mapping data is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device, and storage medium for generating mapping data, so as to reduce the cost of the robot while generating accurate mapping data. The specific technical solutions are as follows:

[0006] The embodiments of the present application first provide a method for generating mapping data, and the method includes:

[0007] Obtain each to-be-utilized image collected by the image acquisition device of the robot when the robot travels in the scene to be mapped, and the to-be-utilized pose corresponding to each to-be-utilized image; wherein, the to-be-utilized pose corresponding to each to-be-utilized image is: the pose of the robot collected by the odometer of the robot when collecting this to-be-utilized image;

[0008] Based on the to-be-utilized poses corresponding to each to-be-utilized image and the pixel coordinates of the valid feature points in each to-be-utilized image, calculate the three-dimensional coordinates of the real positions represented by each valid feature point; wherein, any valid feature point has other feature points with feature matching, and the to-be-utilized image to which the other feature points with feature matching belong is adjacent to the to-be-utilized image to which this valid feature point belongs;

[0009] Based on the pixel coordinates of the feature points with feature matching in every two to-be-utilized images having a loop closure relationship, and the three-dimensional coordinates of the real positions represented by this feature point with feature matching, calculate the pose change of the robot when collecting these two to-be-utilized images, and obtain the relative loop closure observation pose between these two to-be-utilized images;

[0010] Taking the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image, the association relationship between each effective feature point and the represented real position, and the relative loop-closure observation pose between the to-be-utilized images with loop-closure relationships as constraint conditions, the data of the to-be-utilized poses corresponding to each to-be-utilized image and the three-dimensional coordinates of the real positions represented by each effective feature point are optimized to obtain the final true pose of the robot when each to-be-utilized image is acquired and the final true coordinates of the real positions represented by each effective feature point, which are used as the current mapping data.

[0011] Optionally, the taking the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image, the association relationship between each effective feature point and the represented real position, and the relative loop-closure observation pose between the to-be-utilized images with loop-closure relationships as constraint conditions, and optimizing the data of the to-be-utilized poses corresponding to each to-be-utilized image and the three-dimensional coordinates of the real positions represented by each effective feature point to obtain the final true pose of the robot when each to-be-utilized image is acquired and the final true coordinates of the real positions represented by each effective feature point includes:

[0012] Constructing a first constraint function, a second constraint function, and a third constraint function; wherein, the first constraint function represents the difference between the pose difference between the true poses of the robot when every two to-be-utilized images are acquired and the pose difference between the to-be-utilized poses corresponding to these two to-be-utilized images; the second constraint function represents the difference between the pixel coordinates projected by the true coordinates of the real position represented by each effective feature point in the to-be-utilized image to which it belongs and the pixel coordinates of this effective feature point in the to-be-utilized image to which it belongs; the third constraint function represents the difference between the pose difference between the true poses of the robot when every two to-be-utilized images with loop-closure relationships are acquired and the loop-closure pose difference; the loop-closure pose difference represents the difference between the poses of the robot when these two to-be-utilized images are acquired, determined based on the feature points with feature matches in these two to-be-utilized images.

[0013] Taking the minimum sum of the first constraint function, the second constraint function, and the third constraint function as the optimization objective, the data of the to-be-utilized poses corresponding to each to-be-utilized image and the three-dimensional coordinates of the real positions represented by each effective feature point are optimized to obtain the final true pose of the robot when each to-be-utilized image is acquired and the final true coordinates of the real positions represented by each effective feature point.

[0014] Optionally, using the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image, the association relationship between each effective feature point and the represented real position, and the loop observation relative pose between the to-be-utilized images with loop relationship as constraint conditions, data optimization is performed on the to-be-utilized poses corresponding to each to-be-utilized image and the three-dimensional coordinates of the real positions represented by each effective feature point, to obtain the final true pose of the robot when each to-be-utilized image is acquired and the final true coordinates of the real positions represented by each effective feature point, including:

[0015] Using the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image and the association relationship between each effective feature point and the represented real position as constraint conditions, data optimization is performed on the to-be-utilized poses corresponding to each to-be-utilized image, to obtain the intermediate true pose of the robot when each to-be-utilized image is acquired;

[0016] Using the change relationship between the intermediate true poses corresponding to each to-be-utilized image and the loop observation relative pose between the to-be-utilized images with loop relationship as constraint conditions, data optimization is performed on the intermediate true poses corresponding to each to-be-utilized image, to obtain the final true pose of the robot when each to-be-utilized image is acquired;

[0017] Based on the final true pose of the robot when each to-be-utilized image is acquired, and the pixel coordinates of the effective feature points representing the same real position in each to-be-utilized image, the final true coordinates of this real position are calculated.

[0018] Optionally, the step of using the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image and the association relationship between each effective feature point and the represented real position as constraint conditions, and performing data optimization on the to-be-utilized poses corresponding to each to-be-utilized image to obtain the intermediate true pose of the robot when each to-be-utilized image is acquired, includes:

[0019] Select a first specified number of to-be-utilized images from the currently unselected to-be-utilized images;

[0020] Using the change relationship between the to-be-utilized poses corresponding to the selected to-be-utilized images and the association relationship between each effective feature point in the selected to-be-utilized images and the represented real position as constraint conditions, data optimization is performed on the to-be-utilized poses corresponding to the selected to-be-utilized images, to obtain the intermediate true pose of the robot when each selected to-be-utilized image is acquired;

[0021] Return to execute the step of selecting a first specified number of to-be-utilized images from the currently unselected to-be-utilized images until the intermediate true poses corresponding to each to-be-utilized image are obtained.

[0022] Optionally, using the change relationship between the to-be-utilized poses corresponding to the selected to-be-utilized images and the association relationship between each valid feature point in the selected to-be-utilized images and the represented real positions as constraint conditions, data optimization is performed on the to-be-utilized poses corresponding to the selected to-be-utilized images to obtain the intermediate true poses of the robot when each selected to-be-utilized image is collected, including:

[0023] Construct a fourth constraint function and a fifth constraint function; wherein, the fourth constraint function represents: the difference between the pose differences between the true poses of the robot when collecting every two selected to-be-utilized images and the pose differences between the to-be-utilized poses corresponding to these two to-be-utilized images; the fifth constraint function represents: the difference between the pixel coordinates projected in the to-be-utilized image of the real coordinates of the real position represented by each valid feature point in the selected to-be-utilized image and the pixel coordinates of this valid feature point in the to-be-utilized image to which it belongs;

[0024] Taking the minimum sum of the fourth constraint function and the fifth constraint function as the optimization objective, data optimization is performed on the to-be-utilized poses corresponding to the selected to-be-utilized images to obtain the intermediate true poses of the robot when each selected to-be-utilized image is collected;

[0025] Using the change relationship between the intermediate true poses corresponding to each to-be-utilized image and the relative loop-closure observation poses between the to-be-utilized images with loop-closure relationships as constraint conditions, data optimization is performed on the intermediate true poses corresponding to each to-be-utilized image to obtain the final true poses of the robot when each to-be-utilized image is collected, including:

[0026] Construct a third constraint function and a sixth constraint function; wherein, the third constraint function represents: the difference between the pose differences between the true poses of the robot when collecting every two to-be-utilized images with loop-closure relationships and the loop-closure pose difference; the loop-closure pose difference represents: the difference between the poses of the robot when collecting these two to-be-utilized images determined based on the feature points with feature matches in these two to-be-utilized images; the sixth constraint function represents: the difference between the pose differences between the true poses of the robot when collecting every two to-be-utilized images and the pose differences between the intermediate true poses of the robot when collecting these two to-be-utilized images;

[0027] Taking the minimum sum of the third constraint function and the sixth constraint function as the optimization objective, data optimization is performed on the intermediate true poses corresponding to each to-be-utilized image to obtain the final true poses of the robot when each to-be-utilized image is collected.

[0028] Optionally, before calculating the pose change of the robot when collecting the two to-be-utilized images based on the pixel coordinates of the feature points that match in the to-be-utilized images with a loop relationship for every two of them and the three-dimensional coordinates of the real positions represented by the feature points that match, the method further includes:

[0029] For each to-be-utilized image, determine, from other to-be-utilized images before the collection time of this to-be-utilized image, the image whose corresponding to-be-utilized pose satisfies position loop with this to-be-utilized image as the coarsely screened image corresponding to this to-be-utilized image; wherein, the distance between the to-be-utilized poses corresponding to two images that satisfy position loop is less than the first distance threshold, and / or, the total distance that the odometer has collected for the robot to move between the collection times of two images that satisfy position loop is greater than the second distance threshold;

[0030] Determine, from the coarsely screened images corresponding to this to-be-utilized image, the image whose number of feature points that match with this to-be-utilized image is greater than the preset quantity threshold as the image having a loop relationship with this to-be-utilized image.

[0031] Optionally, the determining, from the coarsely screened images corresponding to this to-be-utilized image, the image whose number of feature points that match with this to-be-utilized image is greater than the preset quantity threshold as the image having a loop relationship with this to-be-utilized image includes:

[0032] Determine, from the coarsely screened images corresponding to this to-be-utilized image, the image whose global image feature similarity with this to-be-utilized image is greater than the preset similarity threshold as the similar image corresponding to this to-be-utilized image;

[0033] Determine, from the similar images corresponding to this to-be-utilized image, the image whose number of feature points that match with this to-be-utilized image is greater than the preset quantity threshold as the image having a loop relationship with this to-be-utilized image.

[0034] Optionally, obtain the to-be-utilized pose corresponding to each to-be-utilized image through the following steps:

[0035] When the pose of the robot changes by a specified angle, perform motion estimation based on the images collected by the image acquisition device, and determine the timestamp of the image collected when the robot undergoes the specified angle change as the first timestamp; and based on the pose collected by the odometer, determine the timestamp of the pose collected by the odometer when the robot undergoes the angle change as the second timestamp;

[0036] Perform time alignment on the timestamps of each to-be-utilized image and the timestamps of the poses collected by the odometer based on the time difference between the first timestamp and the second timestamp;

[0037] Interpolate the collected poses according to the timestamps of the to-be-utilized images after alignment and the timestamps of the poses after alignment, so that the timestamps of the interpolated poses are the same as the timestamps of the to-be-utilized images after alignment, and obtain the poses of the robot collected by the odometer of the robot when each to-be-utilized image is collected, as the to-be-utilized poses corresponding to the to-be-utilized images.

[0038] Optionally, each to-be-utilized image is obtained by frame extraction from the images collected when the robot travels in the to-be-mapped scene; the difference between the to-be-utilized poses corresponding to every two adjacent to-be-utilized images is greater than a preset pose difference.

[0039] Optionally, the method further includes:

[0040] For each valid feature point, calculate the average value of the feature similarities between the valid feature point and the valid feature points that match other features, as the average similarity of the valid feature point;

[0041] For each real position represented by the valid feature points, use the valid feature point with the minimum average similarity among the valid feature points representing the real position as the stable feature point of the real position, and record the features of the determined stable feature points and the association relationships between the stable feature points and the to-be-utilized images in the current mapping data; where, the images and stable feature points with association relationships indicate that the image contains the associated stable feature points or feature points that feature-match the associated stable feature points.

[0042] Optionally, the current mapping data further contains the global image features of the to-be-utilized images. After obtaining the current mapping data, the method further includes:

[0043] Extract the global image features and the feature points included in the to-be-localized image;

[0044] Determine the to-be-utilized image whose global image features match the global image features of the to-be-localized image as the matching image of the to-be-localized image;

[0045] Determine the stable feature points that feature-match the feature points included in the to-be-localized image from the stable feature points associated with the matching image of the to-be-localized image;

[0046] Calculate the pose of the robot when the to-be-localized image is collected based on the final true coordinates of the real positions represented by the determined stable feature points, as the localization result of the to-be-localized image.

[0047] Optionally, the current mapping data further contains the global image features of the to-be-utilized images, and the method further includes:

[0048] Obtain historical mapping data; wherein, the historical mapping data includes: global image features of historical collected images, features of historically determined stable feature points, and final true coordinates of the real positions represented by the historically determined stable feature points; the association relationship between historical collected images and stable feature points;

[0049] Perform feature matching on the global image features of each image to be utilized and the global image features of historical collected images to obtain the images to be utilized and historical collected images that match;

[0050] For the images to be utilized and historical collected images that match, perform feature matching on the stable feature points associated with the image to be utilized and the stable feature points associated with the historical collected image to obtain matching stable feature points;

[0051] Take the matching stable feature points as stable feature points representing the same real position, and merge the current mapping data and the historical mapping data.

[0052] Optionally, the method further includes:

[0053] Generate a mapping map based on the current mapping data and display it;

[0054] When receiving an editing instruction for the generated mapping map, perform an editing operation on the generated mapping map according to the editing instruction;

[0055] The editing operation includes at least one of the following:

[0056] Perform overall translation and rotation on the coordinates of each pose and each real position included in the current mapping map;

[0057] Delete at least one pose included in the current mapping map;

[0058] Delete the coordinates of at least one real position included in the current mapping map.

[0059] Optionally, the features of each feature point, and / or the global image features of each image are obtained by using a deep learning network model.

[0060] An embodiment of the present application further provides a device for generating mapping data, and the device includes:

[0061] An information acquisition module, configured to acquire each image to be utilized collected by an image acquisition device of the robot when the robot travels in a scene to be mapped, and the corresponding poses to be utilized of each image to be utilized; wherein, the pose to be utilized corresponding to each image to be utilized is: the pose of the robot collected by the odometer of the robot when collecting the image to be utilized;

[0062] A three-dimensional coordinate calculation module, which is used to calculate the three-dimensional coordinates of the real positions represented by each valid feature point based on the to-be-utilized poses corresponding to each to-be-utilized image and the pixel coordinates of the valid feature points in each to-be-utilized image; wherein, there are other feature points that are feature-matched to any valid feature point, and the to-be-utilized image to which the other feature points that are feature-matched belong is adjacent to the to-be-utilized image to which this valid feature point belongs.

[0063] A loop observation relative pose determination module, which is used to calculate the pose change of the robot when collecting the two to-be-utilized images based on the pixel coordinates of the feature points that are feature-matched in every two to-be-utilized images with a loop relationship and the three-dimensional coordinates of the real positions represented by the feature points that are feature-matched, so as to obtain the loop observation relative pose between the two to-be-utilized images.

[0064] A data optimization module, which is used to optimize the data of the to-be-utilized poses corresponding to each to-be-utilized image and the three-dimensional coordinates of the real positions represented by each valid feature point by taking the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image, the association relationship between each valid feature point and the represented real position, and the loop observation relative pose between the to-be-utilized images with a loop relationship as constraint conditions, so as to obtain the final true pose of the robot when collecting each to-be-utilized image and the final true coordinates of the real positions represented by each valid feature point, which are used as the current mapping data.

[0065] Optionally, the data optimization module includes:

[0066] A first function construction sub-module, which is used to construct a first constraint function, a second constraint function and a third constraint function; wherein, the first constraint function represents the difference between the pose difference between the true poses of the robot when collecting every two to-be-utilized images and the pose difference between the to-be-utilized poses corresponding to the two to-be-utilized images; the second constraint function represents the difference between the pixel coordinates projected by the true coordinates of the real position represented by each valid feature point in the to-be-utilized image to which it belongs and the pixel coordinates of this valid feature point in the to-be-utilized image to which it belongs; the third constraint function represents the difference between the pose difference between the true poses of the robot when collecting every two to-be-utilized images with a loop relationship and the loop pose difference; the loop pose difference represents the difference between the poses of the robot when collecting the two to-be-utilized images determined based on the feature points that are feature-matched in the two to-be-utilized images.

[0067] The first data optimization sub-module is used to optimize the data of the to-be-utilized poses corresponding to each to-be-utilized image and the three-dimensional coordinates of the real positions represented by each valid feature point with the goal of minimizing the sum of the first constraint function, the second constraint function, and the third constraint function, so as to obtain the final true pose of the robot when collecting each to-be-utilized image and the final true coordinates of the real positions represented by each valid feature point;

[0068] and / or,

[0069] The data optimization module includes:

[0070] The intermediate true pose determination sub-module is used to optimize the data of the to-be-utilized poses corresponding to each to-be-utilized image with the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image and the association relationship between each valid feature point and the represented real position as constraint conditions, so as to obtain the intermediate true pose of the robot when collecting each to-be-utilized image;

[0071] The final true pose determination sub-module is used to optimize the data of the intermediate true poses corresponding to each to-be-utilized image with the change relationship between the intermediate true poses corresponding to each to-be-utilized image and the relative pose of loop observation between the to-be-utilized images with loop relationship as constraint conditions, so as to obtain the final true pose of the robot when collecting each to-be-utilized image;

[0072] The final true coordinate determination sub-module is used to calculate the final true coordinates of the real position based on the final true pose of the robot when collecting each to-be-utilized image and the pixel coordinates of the valid feature points representing the same real position in each to-be-utilized image;

[0073] and / or,

[0074] The intermediate true pose determination sub-module includes:

[0075] The image selection unit is used to select the first specified number of to-be-utilized images from the currently unselected to-be-utilized images;

[0076] The intermediate true pose determination unit is used to optimize the data of the to-be-utilized poses corresponding to the selected to-be-utilized images with the change relationship between the to-be-utilized poses corresponding to the selected to-be-utilized images and the association relationship between each valid feature point in the selected to-be-utilized images and the represented real position as constraint conditions, so as to obtain the intermediate true pose of the robot when collecting each selected to-be-utilized image; trigger the image selection unit to execute the step of selecting the first specified number of to-be-utilized images from the currently unselected to-be-utilized images until the intermediate true poses corresponding to each to-be-utilized image are obtained;

[0077] and / or,

[0078] The intermediate true pose determination unit includes:

[0079] A function construction subunit for constructing a fourth constraint function and a fifth constraint function; wherein, the fourth constraint function represents the difference between the pose difference between the true poses of the robot when collecting every two selected images to be utilized and the pose difference between the corresponding poses to be utilized of the two images to be utilized; the fifth constraint function represents the difference between the pixel coordinates projected in the image to be utilized of the true coordinates of the real position represented by each valid feature point in the selected image to be utilized and the pixel coordinates of the valid feature point in the image to be utilized to which it belongs;

[0080] An intermediate true pose determination subunit for optimizing the data of the poses to be utilized corresponding to the selected images to be utilized with the objective of minimizing the sum of the fourth constraint function and the fifth constraint function, and obtaining the intermediate true pose of the robot when collecting each selected image to be utilized;

[0081] The final true pose determination sub-module includes:

[0082] A function construction unit for constructing a third constraint function and a sixth constraint function; wherein, the third constraint function represents the difference between the pose difference between the true poses of the robot when collecting every two images to be utilized with a loop closure relationship and the loop closure pose difference; the loop closure pose difference represents the difference between the poses of the robot when collecting the two images to be utilized determined based on the feature points with feature matches in the two images to be utilized; the sixth constraint function represents the difference between the pose difference between the true poses of the robot when collecting every two images to be utilized and the pose difference between the intermediate true poses of the robot when collecting the two images to be utilized;

[0083] A final true pose determination unit for optimizing the data of the intermediate true poses corresponding to each image to be utilized with the objective of minimizing the sum of the third constraint function and the sixth constraint function, and obtaining the final true pose of the robot when collecting each image to be utilized;

[0084] And / or, the device further includes:

[0085] A coarse screening module, which is used to, before the loop observation relative pose determination module calculates the pose change of the robot when collecting the two images to be utilized based on the pixel coordinates of the feature points that are feature-matched in each pair of images to be utilized with a loop relationship and the three-dimensional coordinates of the real positions represented by the feature-matched feature points, and obtains the loop observation relative pose between the two images to be utilized, for each image to be utilized, determine, from other images to be utilized before the acquisition time of this image to be utilized, an image whose corresponding pose to be utilized satisfies position loop with this image to be utilized as the coarse screening image corresponding to this image to be utilized; wherein, the distance between the corresponding poses to be utilized of two images that satisfy position loop is less than the first distance threshold, and / or, the total distance that the odometer has collected for the robot to move between the acquisition times of two images that satisfy position loop is greater than the second distance threshold;

[0086] A loop determination module, which is used to determine, from the coarse screening images corresponding to this image to be utilized, an image whose number of feature points that are feature-matched with this image to be utilized is greater than the preset quantity threshold as the image having a loop relationship with this image to be utilized;

[0087] and / or,

[0088] The loop determination module includes:

[0089] A similar image determination sub-module, which is used to determine, from the coarse screening images corresponding to this image to be utilized, an image whose global image feature similarity with this image to be utilized is greater than the preset similarity threshold as the similar image corresponding to this image to be utilized;

[0090] A loop determination sub-module, which is used to determine, from the similar images corresponding to this image to be utilized, an image whose number of feature points that are feature-matched with this image to be utilized is greater than the preset quantity threshold as the image having a loop relationship with this image to be utilized;

[0091] and / or,

[0092] The information acquisition module includes:

[0093] A timestamp determination sub-module, which is used, when the pose of the robot undergoes a specified angular change, to perform motion estimation based on the images collected by the image acquisition device, determine the timestamp of the image collected when the robot undergoes the specified angular change as the first timestamp; and determine the timestamp of the pose collected by the odometer when the robot undergoes the angular change as the second timestamp based on the pose collected by the odometer;

[0094] A time alignment sub-module, configured to perform time alignment on the timestamps of each image to be utilized and the timestamps of the poses collected by the odometer based on the time difference between the first timestamp and the second timestamp;

[0095] An interpolation sub-module, configured to perform interpolation on the collected poses according to the timestamps of each image to be utilized after alignment and the timestamps of the poses after alignment, so that the timestamps of the interpolated poses are the same as the timestamps of each image to be utilized after alignment, and obtain the poses of the robot collected by the odometer of the robot when each image to be utilized is collected, as the poses to be utilized corresponding to the image to be utilized;

[0096] and / or

[0097] Each image to be utilized is obtained by extracting frames from the images collected when the robot travels in the scene to be mapped; the difference between the poses to be utilized corresponding to every two adjacent images to be utilized is greater than a preset pose difference;

[0098] and / or

[0099] The device further includes:

[0100] An average similarity calculation module, configured to calculate the average of the feature similarities between each valid feature point and other valid feature points that match the feature, as the average similarity of the valid feature point;

[0101] A stable point determination module, configured to, for each real position represented by a valid feature point, use the valid feature point with the smallest average similarity among the valid feature points representing the real position as the stable feature point of the real position, and record the features of the determined stable feature points and the association relationships between the stable feature points and the images to be utilized in the current mapping data; wherein, the images and stable feature points with an association relationship mean that the image contains the associated stable feature points or feature points that feature-match the associated stable feature points;

[0102] and / or

[0103] The current mapping data further contains the global image features of the images to be utilized, and the device further includes:

[0104] A to-be-localized image processing module, configured to extract the global image features and the feature points included in the to-be-localized image after generating the current mapping data;

[0105] A matching image determination module, configured to determine the image to be utilized whose global image features match the global image features of the to-be-localized image as the matching image of the to-be-localized image;

[0106] A matching feature point determination module, configured to determine, from the stable feature points associated with the matching image of the to-be-localized image, the stable feature points that feature-match the feature points included in the to-be-localized image;

[0107] A pose calculation module, configured to calculate the pose of the robot when the to-be-localized image is acquired, based on the final true coordinates of the real positions represented by the determined stable feature points, as the localization result of the to-be-localized image;

[0108] And / or,

[0109] The current mapping data further includes the global image features of the to-be-utilized image, and the apparatus further includes:

[0110] A historical mapping data acquisition module, configured to acquire historical mapping data; wherein, the historical mapping data includes: the global image features of historical acquired images, the features of historically determined stable feature points, the final true coordinates of the real positions represented by the historically determined stable feature points; the association relationship between historical acquired images and stable feature points;

[0111] A global image feature matching module, configured to perform feature matching on the global image features of each to-be-utilized image and the global image features of historical acquired images, to obtain the to-be-utilized images and historical acquired images that match each other;

[0112] A stable feature point matching module, configured to, for the to-be-utilized image and historical acquired image that match each other, perform feature matching on the stable feature points associated with the to-be-utilized image and the stable feature points associated with the historical acquired image, to obtain the stable feature points that match each other;

[0113] A merging module, configured to use the matched stable feature points as the stable feature points representing the same real position, and merge the current mapping data and the historical mapping data;

[0114] And / or,

[0115] The apparatus further includes:

[0116] A mapping map generation module, configured to generate and display a mapping map based on the current mapping data;

[0117] An editing module, configured to, when receiving an editing instruction for the generated mapping map, perform an editing operation on the generated mapping map according to the editing instruction; the editing operation includes at least one of the following:

[0118] Perform overall translation and rotation on the coordinates of each pose and each real position included in the current mapping map;

[0119] Delete at least one pose included in the current mapping map;

[0120] Delete the coordinates of at least one real position included in the current mapping map;

[0121] And / or,

[0122] The features of each feature point and / or the global image features of each image are obtained by using a deep learning network model.

[0123] An embodiment of the present application further provides an electronic device, including:

[0124] A memory for storing a computer program;

[0125] A processor for implementing the method for generating mapping data described in any one of the above when executing the program stored on the memory.

[0126] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, and the computer program realizes the method for generating mapping data described in any one of the above when executed by a processor.

[0127] An embodiment of the present application further provides a computer program product including instructions, which when running on a computer, causes the computer to execute the method for generating mapping data described in any one of the above.

[0128] Advantageous effects of the embodiments of the present application:

[0129] The method for determining mapping data provided by the embodiments of the present application obtains the to-be-used images collected when the robot travels in the to-be-mapped scene and the to-be-used poses determined based on the odometer, and then uses the change relationship between the to-be-used poses corresponding to the to-be-used images, the pixel coordinates of each effective feature point in the to-be-used image to which it belongs, and the loop observation relative pose between the to-be-used images with a loop relationship as constraint conditions to optimize the data, and obtains the final true pose of the robot when each to-be-used image is collected and the final true coordinates of the real positions represented by each effective feature point. It can be seen that this solution optimizes the to-be-used image corresponding pose and the real position represented by the effective feature points included in the to-be-used image by using a data optimization method, can reduce the error of the finally determined true pose of the robot and the real positions represented by each effective feature point, improve the accuracy of the mapping data, and only two sensors, an image acquisition device and an odometer, need to be set in the robot of this solution to generate accurate mapping data. Therefore, through this solution, the cost of the robot can be reduced while accurate mapping data is generated.

[0130] Of course, it is not necessary for any product or method implementing the present application to achieve all the above advantages at the same time. Description of the Drawings

[0131] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other embodiments can also be obtained based on these drawings.

[0132] Figure 1 The first flowchart of the method for generating mapping data provided by the embodiment of the present application;

[0133] Figure 2 The schematic diagram of map optimization in the method for generating mapping data provided by the embodiment of the present application;

[0134] Figure 3 The second flowchart of the method for generating mapping data provided by the embodiment of the present application;

[0135] Figure 4 The third flowchart of the method for generating mapping data provided by the embodiment of the present application;

[0136] Figure 5 The fourth flowchart of the method for generating mapping data provided by the embodiment of the present application;

[0137] Figure 6 The fifth flowchart of the method for generating mapping data provided by the embodiment of the present application;

[0138] Figure 7 The sixth flowchart of the method for generating mapping data provided by the embodiment of the present application;

[0139] Figure 8 The structural schematic diagram of the device for generating mapping data provided by the embodiment of the present application;

[0140] Figure 9 The structural schematic diagram of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0141] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0142] In order to reduce the cost of the robot while generating accurate mapping data, the embodiments of the present application provide a method, an apparatus, an electronic device, and a storage medium for generating mapping data. Among them, the method for generating mapping data can be applied to an electronic device, such as a computer, a server, etc. In a specific application, the method can also be applied to a robot with data processing capabilities, and the robot can be provided with an image acquisition device and an odometer. The method can include the following steps:

[0143] Obtain each to-be-utilized image collected by the image acquisition device of the robot and the to-be-utilized pose corresponding to each to-be-utilized image when the robot travels in the to-be-mapped scene; wherein, the to-be-utilized pose corresponding to each to-be-utilized image is: the pose of the robot collected by the odometer of the robot when collecting the to-be-utilized image;

[0144] Based on the to-be-utilized poses corresponding to each to-be-utilized image and the pixel coordinates of the valid feature points in each to-be-utilized image, calculate the three-dimensional coordinates of the real positions represented by each valid feature point; wherein, any valid feature point has other feature points with feature matching, and the to-be-utilized image to which the other feature points with feature matching belong is adjacent to the to-be-utilized image to which the valid feature point belongs;

[0145] Based on the pixel coordinates of the feature points with feature matching in every two to-be-utilized images with a loop closure relationship and the three-dimensional coordinates of the real positions represented by the feature points with feature matching, calculate the pose change of the robot when collecting the two to-be-utilized images, and obtain the relative loop closure observation pose between the two to-be-utilized images;

[0146] Taking the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image, the association relationship between each valid feature point and the represented real position, and the relative loop closure observation pose between the to-be-utilized images with a loop closure relationship as constraint conditions, optimize the to-be-utilized poses corresponding to each to-be-utilized image and the three-dimensional coordinates of the real positions represented by each valid feature point to obtain the final true pose of the robot when collecting each to-be-utilized image and the final true coordinates of the real positions represented by each valid feature point, as the current mapping data.

[0147] In this embodiment, a data optimization method is used to optimize the poses corresponding to the to-be-utilized images and the real positions represented by the valid feature points included in the to-be-utilized images, which can reduce the errors of the finally determined true pose of the robot and the real positions represented by each valid feature point, improve the accuracy of the mapping data, and only two sensors, namely an image acquisition device and an odometer, need to be set in the robot of this solution to generate accurate mapping data. Therefore, through this solution, accurate mapping data can be generated while reducing the cost of the robot.

[0148] The following provides an exemplary introduction to the method for generating mapping data provided in the embodiments of the present application with reference to the accompanying drawings. As Figure 1 shown, the method for generating the mapping data may include the following steps:

[0149] S101. Obtain each image to be utilized collected by the image acquisition device of the robot and the corresponding pose to be utilized when the robot travels in the scene to be mapped; wherein, the pose to be utilized corresponding to each image to be utilized is: the pose of the robot collected by the odometer of the robot when collecting the image to be utilized.

[0150] The pose of the robot may include the position of the robot, such as the coordinates of the robot in a preset world coordinate system, and the orientation of the robot. The odometer may be a wheel odometer, an inertial measurement unit (IMU), etc. To save costs, the odometer may be a wheel odometer. By setting an odometer for the robot, the pose of the robot can be collected during the travel of the robot, and then, for each image to be utilized collected by the image acquisition device of the robot, the pose that is the same as the acquisition moment of the image to be utilized can be determined from the poses collected by the odometer as the pose to be utilized corresponding to the image to be utilized. The image acquisition device of the robot may be a monocular camera, and of course, it is not limited thereto.

[0151] The collected images to be utilized may be all the images collected by the robot when traveling in the scene to be mapped, or may be obtained by frame extraction from the images collected by the robot when traveling in the scene to be mapped, and the difference between the poses to be utilized corresponding to every two adjacent images to be utilized is greater than a preset pose difference. The preset pose difference may include a preset distance difference threshold and / or a preset angle difference threshold. For example, the preset distance difference threshold may be 0.5 m, and the preset angle difference threshold may be 10°. If the difference between the poses of the robot when collecting two images is greater than the preset pose difference, it indicates that there is a certain difference between these two images. By this way of frame extraction, data redundancy can be reduced, the amount of data of the images to be processed subsequently can be decreased, and the processing speed can be improved.

[0152] S102. Calculate the three-dimensional coordinates of the real positions represented by each valid feature point based on the pose to be utilized corresponding to each image to be utilized and the pixel coordinates of the valid feature points in each image to be utilized; wherein, there are other feature points that are feature-matched with any valid feature point, and the image to be utilized to which the other feature points that are feature-matched belong is adjacent to the image to be utilized to which the valid feature point belongs.

[0153] Feature points in the image can be obtained by processing the image using a feature point extraction algorithm. The feature point extraction algorithm can be the SIFT (Scale-Invariant Feature Transform) algorithm, the SURF (Speeded-Up Robust Feature) algorithm, etc. Through the feature point extraction algorithm, the pixel coordinates of the feature points in the image and the features of the feature points, also known as feature descriptors, can be determined. To improve the robustness of the feature points to displacement, rotation, deformation, and even illumination changes to be applicable to complex scenarios, a deep learning network model can be used to extract the feature points and the features of the feature points in the image. The deep learning network model can be the SuperPoint model, the DISK (Learning Local Features with Policy Gradients) model, etc. In addition, the RANSAC (Random Sample Consensus) algorithm can be used to match the feature points in every two images.

[0154] In this embodiment, the feature points that are feature-matched in two adjacent images to be utilized, the poses to be utilized corresponding to the two images to be utilized, and the pixel coordinates of the feature points that are feature-matched in the two images to be utilized can be used for triangulation calculation to determine the three-dimensional coordinates of the real position represented by the feature points. Therefore, for each feature point in each image to be utilized, if there is no matching relationship between this feature point and the adjacent image to be utilized, it indicates that this feature point cannot be triangulated and can be regarded as an invalid feature point. In this way, the valid feature points can be determined from all the feature points.

[0155] To avoid false matching, in this embodiment, the feature points that are continuously matched successfully can also be used as valid feature points. For example, if there is a feature point a1 in an image to be utilized, and there are feature points that are feature-matched with feature point a1 in a specified number of other adjacent images to be utilized, then feature point a1 can be determined as a valid feature point. For example, the specified number can be 5, and the acquisition times of the other adjacent images to be utilized can be before or after the image to be utilized.

[0156] After determining the effective feature points, the triangulation algorithm can be used to calculate the three-dimensional coordinates of the real positions represented by each effective feature point based on the poses to be utilized corresponding to each image to be utilized and the pixel coordinates of the effective feature points in each image to be utilized, so as to obtain the initial mapping data. At this time, the obtained three-dimensional coordinates may have relatively large errors. Therefore, subsequent data optimization can be performed on them, and the specific process will be introduced below. Further, if the three-dimensional coordinates of the real position represented by a calculated feature point exceed the image field of view range of the image to be utilized to which the effective feature point belongs, it indicates that the feature point may be a mismatched point, and this feature point can be removed from the effective feature points. In addition, if the distance between the pixel positions of the feature-matched feature points in two adjacent images to be utilized is small, that is, less than the preset pixel distance threshold, since the error is large when calculating the three-dimensional coordinates of the real position based on these two feature points, they can also be removed from the effective feature points.

[0157] Further, after this step, the association relationship between each effective feature point and the represented real position can also be established, and the change relationship between the poses to be utilized corresponding to each image to be utilized can be determined, so as to facilitate subsequent data optimization of the poses to be utilized corresponding to each image to be utilized and the three-dimensional coordinates of the real positions represented by each effective feature point. Among them, the change relationship between the poses to be utilized corresponding to each image to be utilized can be the pose difference between the poses to be utilized corresponding to every two images to be utilized. In order to reduce the computational amount, only the pose differences between the images to be utilized adjacent at every two acquisition times can also be determined. The specific process of data optimization will be introduced below.

[0158] S103, based on the pixel coordinates of the feature-matched feature points in every two images to be utilized with a loop closure relationship and the three-dimensional coordinates of the real positions represented by the feature-matched feature points, calculate the pose change of the robot when collecting these two images to be utilized, and obtain the relative loop closure observation pose between these two images to be utilized;

[0159] Among them, loop closure means that the robot returns to a place it has been to during the movement process. The images to be utilized with a loop closure relationship are the images to be utilized collected when the robot passes through the same place twice.

[0160] In one implementation, to determine the images to be utilized with loop relationships, for each image to be utilized, the image to be utilized can be matched with other images to be utilized before the acquisition time to determine the images with the number of feature points that match the feature points of the image to be utilized being greater than a preset quantity threshold as the images having loop relationships with the image to be utilized. If there is no image among other images to be utilized with the number of feature points that match the feature points of the image to be utilized being greater than the preset quantity threshold, it indicates that there is no loop relationship between each of the other images to be utilized and the image to be utilized. Among them, the preset quantity threshold can be set according to the number of feature points that can be extracted from the image. For example, if the number of feature points that can be extracted from the image is around 500, the preset quantity threshold can be set to a value between 200 and 300, and can be specifically set according to experience.

[0161] In one implementation, before step S103, the images to be utilized with loop relationships can be determined in the following manner:

[0162] Step A1, for each image to be utilized, determine, from other images to be utilized before the acquisition time of this image to be utilized, the images whose corresponding poses to be utilized satisfy position loop as the roughly screened images corresponding to this image to be utilized; among them, the distance between the corresponding poses to be utilized of two images that satisfy position loop is less than the first distance threshold, and / or, the total distance of the robot movement collected by the odometer between the acquisition times of two images that satisfy position loop is greater than the second distance threshold;

[0163] The above total distance refers to: the cumulative distance of the robot movement collected by the odometer during the interval time of collecting two images that satisfy position loop. Since the robot may have turned during this interval time, this total distance is not necessarily the relative distance between the corresponding positions of these two images. It can be understood that during the interval time of collecting two images to be utilized, the cumulative distance of the robot movement should be long enough, and the relative distance of the robot when collecting these two images to be utilized should be close enough. Such two images to be utilized are more likely to be images with loop relationships. In this embodiment, for each image to be utilized, first determine the images to be utilized whose poses satisfy position loop to preliminarily screen other images to be utilized and obtain the roughly screened images corresponding to this image to be utilized.

[0164] Step A2, determine, from the roughly screened images corresponding to this image to be utilized, the images with the number of feature points that match the feature points of this image to be utilized being greater than the preset quantity threshold as the images having loop relationships with this image to be utilized.

[0165] Since the computational cost of performing feature point matching is relatively large, in this embodiment, the images can be preliminarily screened first to reduce the number of images for which feature point matching needs to be performed, thereby reducing the computational cost in the process of determining the images to be utilized with loop-back relationships and improving the processing speed.

[0166] In one implementation, the feature points of the image to be utilized can be directly matched with the corresponding coarsely screened image to determine the images having loop-back relationships with the image to be utilized.

[0167] In another implementation, the step of determining, from the coarsely screened images corresponding to the image to be utilized, the images for which the number of feature points with feature matching with the image to be utilized is greater than a preset quantity threshold as the images having loop-back relationships with the image to be utilized may include the following steps:

[0168] Step A21: Determine, from the coarsely screened images corresponding to the image to be utilized, the images with a global image feature similarity greater than a preset similarity threshold with the image to be utilized as the similar images corresponding to the image to be utilized;

[0169] The above-mentioned global image features refer to the features that can represent the entire image, such as the overall features that can describe the color and shape of the image or the target, also known as global descriptors. The global image features can be HOG (Histogram of Oriented Gradient) features. The global image features can also be obtained by processing the image using a deep learning network model, and such a model can be a CNN (Convolutional Neural Networks) model, etc.

[0170] It can be understood that when the robot returns to a place it has been to before, there will be a certain similarity between the images collected twice. Therefore, in this step, based on determining the coarsely screened images corresponding to the image to be utilized, the similarity of the global image features between each coarsely screened image and the image to be utilized can be calculated. For example, the cosine distance of the global image features between the coarsely screened image and the image to be utilized can be calculated. Then, the images with a global image feature similarity greater than the preset similarity threshold are determined from the coarsely screened images as the similar images corresponding to the image to be utilized to further screen the coarsely screened images.

[0171] Step A22: Determine, from the similar images corresponding to the image to be utilized, the images for which the number of feature points with feature matching with the image to be utilized is greater than a preset quantity threshold as the images having loop-back relationships with the image to be utilized.

[0172] In this embodiment, by using the similarity of the global image features, the rough screening image corresponding to the image to be used is further screened, further reducing the number of images for which feature point matching is to be performed, so that the computational amount can be further reduced and the processing speed can be improved.

[0173] It can be understood that although two images to be used with a loop relationship are determined, when collecting these two images to be used, the poses of the robot may not be exactly the same. Therefore, in this step, the pixel coordinates of the feature points that are feature-matched in the two images to be used with a loop relationship, and the three-dimensional coordinates of the real positions represented by the matched feature points can be used to calculate the pose change of the robot when collecting these two images to be used. The pose change of the robot can refer to the position change and angle change of the robot. Specifically, for each image to be used in these two images to be used, the PnP (Perspective-n-Point) algorithm can be used to calculate the pose of the robot when collecting the image to be used based on the pixel coordinates of the valid feature points in the image to be used and the three-dimensional coordinates of the real positions represented by the valid feature points. Furthermore, the pose change of the robot when collecting these two images to be used can be calculated as the relative pose of the loop observation.

[0174] S104. Using the change relationship between the poses to be used corresponding to each image to be used, the association relationship between each valid feature point and the represented real position, and the relative pose of the loop observation between the images to be used with a loop relationship as constraint conditions, optimize the data of the poses to be used corresponding to each image to be used and the three-dimensional coordinates of the real positions represented by each valid feature point, and obtain the final true pose of the robot when collecting each image to be used and the final true coordinates of the real positions represented by each valid feature point as the current mapping data.

[0175] In this embodiment, the true pose of the robot when collecting each image to be used and the true coordinates of the real positions represented by the valid feature points can be used as optimization variables, and a constraint function can be constructed according to each constraint condition for data optimization. Since the matched valid feature points actually represent the feature points of the same real position, the true coordinates of the real positions represented by multiple matched valid feature points can be used as one optimization variable.

[0176] Such as Figure 2As shown in the figure, the data optimization process can be regarded as a graph optimization problem. The true pose of the robot when collecting each image to be utilized and the three-dimensional coordinates of the actual positions represented by each valid feature point can be used as the nodes in the graph respectively; the constraint function determined based on the change relationship between the poses to be utilized corresponding to every two images to be utilized is used as the edge between the nodes corresponding to these two images to be utilized; the constraint function determined based on the pixel coordinates of each valid feature point in the image to be utilized to which it belongs is used as the edge between the node corresponding to this valid feature point and the image to be utilized to which it belongs; the constraint function determined based on the relative loop-closure pose of the loop-closure observation between two images to be utilized with a loop-closure relationship (corresponding to the pose of the loop-closure in Figure 2 in the loop-closure) is used as the edge between the nodes corresponding to these two images to be utilized.

[0177] In one implementation, the final true pose of the robot when collecting each image to be utilized and the final true coordinates of the actual positions represented by each valid feature point can be determined through global optimization, which specifically may include:

[0178] Step B1, constructing the first constraint function, the second constraint function, and the third constraint function; among them, the first constraint function represents: the difference between the pose difference between the true poses of the robot when collecting every two images to be utilized and the pose difference between the poses to be utilized corresponding to these two images to be utilized; the second constraint function represents: the difference between the pixel coordinates of the projection of the true coordinates of the actual position represented by each valid feature point in the image to be utilized to which it belongs and the pixel coordinates of this valid feature point in the image to be utilized to which it belongs; the third constraint function represents: the difference between the pose difference between the true poses of the robot when collecting every two images to be utilized with a loop-closure relationship and the loop-closure pose difference; among them, the loop-closure pose difference represents: the difference between the poses of the robot when collecting these two images to be utilized, determined based on the feature points with feature matching in these two images to be utilized;

[0179] The first constraint function can be expressed as: ; among them, represents the pose difference between the true poses of the robot when collecting image to be utilized i and image to be utilized j, that is ; represents the true pose of the robot when collecting image to be utilized i; represents the true pose of the robot when collecting image to be utilized j; , are optimization variables.

[0180] It represents the pose difference between the to-be-utilized pose corresponding to the to-be-utilized image i and the to-be-utilized image j. To reduce the computational load, only the to-be-utilized images adjacent in the acquisition time and the to-be-utilized images with loop closure relationships can be used to construct the first constraint function. That is to say, the to-be-utilized image i and the to-be-utilized image j are the to-be-utilized images adjacent in the acquisition time or the to-be-utilized images with loop closure relationships.

[0181] The second constraint function can be expressed as: ; where K is the camera internal parameter of the image acquisition device, and are optimization variables, represents the true pose of the robot when acquiring the to-be-utilized image i, represents the true coordinates of the real position represented by the valid feature point k in the to-be-utilized image i; it can be understood that, represents the pixel coordinates projected by the true coordinates of the real position represented by the valid feature point k in the to-be-utilized image i, represents the pixel coordinates of the valid feature point k in the to-be-utilized image i.

[0182] The third constraint function can be expressed as: ; where, represents the relative pose of the loop closure observation between the to-be-utilized image i and the to-be-utilized image j with loop closure relationships.

[0183] Step B2: Taking the minimum sum of the first constraint function, the second constraint function, and the third constraint function as the optimization objective, optimize the data of the to-be-utilized pose corresponding to each to-be-utilized image and the three-dimensional coordinates of the real position represented by each valid feature point, and obtain the final true pose of the robot when acquiring each to-be-utilized image and the final true coordinates of the real position represented by each valid feature point.

[0184] That is to say, in this step, the objective function can be constructed based on the sum of the first constraint function, the second constraint function, and the third constraint function. For example, the objective function can be expressed as: .

[0185] After that, through algorithms such as the LM (Levenberg-Marquardt method) algorithm or the Gradient Descent algorithm, taking the to-be-utilized pose corresponding to each to-be-utilized image and the three-dimensional coordinates of the real position represented by each valid feature point as the initial values, calculate the optimal solution of the objective function, and obtain the values of the optimization variables when the objective function reaches the optimal solution, then the final true pose of the robot when acquiring each to-be-utilized image and the final true coordinates of the real position represented by each valid feature point can be obtained.

[0186] In this embodiment, a data optimization method is adopted to optimize the corresponding pose of the image to be utilized and the real positions represented by the valid feature points included in the image to be utilized, which can reduce the errors between the finally determined real pose of the robot and the real positions represented by each valid feature point, improve the accuracy of the mapping data, and only two sensors, namely an image acquisition device and an odometer, need to be set in the robot of this solution to generate accurate mapping data. Therefore, through this solution, accurate mapping data can be generated while reducing the cost of the robot.

[0187] Since the above overall optimization process has a large amount of calculation and occupies a large amount of computing resources, especially a large amount of memory, as the map scale increases, the overall optimization may be difficult to execute. Therefore, in an embodiment of the present application, a step-by-step optimization method can also be adopted. As Figure 3 shown, taking the change relationship between the poses to be utilized corresponding to each image to be utilized, the correlation relationship between each valid feature point and the real position it represents, and the relative pose of the loop observation between the images to be utilized with loop relationship as constraint conditions, data optimization is performed on the poses to be utilized corresponding to each image to be utilized and the three-dimensional coordinates of the real positions represented by each valid feature point to obtain the final real pose of the robot when each image to be utilized is acquired and the final real coordinates of the real positions represented by each valid feature point. The method may include the following steps:

[0188] S1041, taking the change relationship between the poses to be utilized corresponding to each image to be utilized and the correlation relationship between each valid feature point and the real position it represents as constraint conditions, performing data optimization on the poses to be utilized corresponding to each image to be utilized, and obtaining the intermediate real pose of the robot when each image to be utilized is acquired;

[0189] That is to say, in this step, data optimization is first performed using two constraint conditions to obtain an intermediate result of data optimization, that is, the intermediate real pose of the robot. In one implementation, this process may include the following steps:

[0190] Step C1, constructing a first constraint function and a second constraint function; wherein, the first constraint function represents the difference between the pose difference between the real poses of the robot when acquiring every two images to be utilized and the pose difference between the poses to be utilized corresponding to the two images to be utilized; the second constraint function represents the difference between the pixel coordinates projected by the real coordinates of the real position represented by each valid feature point in the image to be utilized to which it belongs and the pixel coordinates of the valid feature point in the image to be utilized to which it belongs;

[0191] The first constraint function and the second constraint function in this step may have the same form as the first constraint function and the second constraint function in the above embodiment, and will not be elaborated here.

[0192] Step C2: Optimize the data of the to-be-utilized poses corresponding to each to-be-utilized image with the objective of minimizing the sum of the first constraint function and the second constraint function, to obtain the intermediate true pose of the robot when collecting each to-be-utilized image.

[0193] In this step, an objective function can be constructed based on the first constraint function and the second constraint function, and this objective function can be expressed as: .

[0194] By solving the optimal solution of this objective function and determining the values of the optimization variables when this objective function reaches the optimal solution, the intermediate true pose of the robot when collecting each to-be-utilized image can be obtained.

[0195] In one implementation, in order to further reduce the computational amount of optimization, the process of optimizing the data of the to-be-utilized poses corresponding to each to-be-utilized image and the three-dimensional coordinates of the real positions represented by each effective feature point with the change relationship between the to-be-utilized poses corresponding to each to-be-utilized image and the correlation relationship between each effective feature point and the represented real position as the constraint conditions to obtain the intermediate true pose of the robot when collecting each to-be-utilized image can also be in a segmented optimization manner, which can specifically include the following steps:

[0196] Step E1: Select the first specified number of to-be-utilized images from the currently unselected to-be-utilized images;

[0197] Among them, the first specified number of to-be-utilized images selected each time can be to-be-utilized images with adjacent acquisition times. For example, in the order of acquisition times, the first specified number of to-be-utilized images with the earliest acquisition times can be selected from the currently unselected to-be-utilized images.

[0198] Step E2: Optimize the data of the to-be-utilized poses corresponding to the selected to-be-utilized images with the change relationship between the to-be-utilized poses corresponding to the selected to-be-utilized images and the correlation relationship between each effective feature point in the selected to-be-utilized images and the represented real position as the constraint conditions, to obtain the intermediate true pose of the robot when collecting each selected to-be-utilized image; return to execute Step E1 until the intermediate true poses corresponding to each to-be-utilized image are obtained.

[0199] In this step, some to-be-utilized images can be selected first, and the true pose of the robot when collecting each selected to-be-utilized image and the true coordinates of the real positions represented by the effective feature points in the selected to-be-utilized images are used as optimization variables for optimization, which can reduce the amount of data to be processed each time of optimization, thereby further reducing the occupation of computing resources.

[0200] Specifically, this process can include:

[0201] Step E21: Construct the fourth constraint function and the fifth constraint function. Among them, the fourth constraint function represents the difference between the pose difference between the true poses of the robot when collecting every two selected images to be utilized and the pose difference between the corresponding poses to be utilized of these two images to be utilized; the fifth constraint function represents the difference between the pixel coordinates of the projection in the image to be utilized of the true coordinates of the real position represented by each valid feature point in the selected image to be utilized and the pixel coordinates of this valid feature point in the image to be utilized to which it belongs.

[0202] Step E22: Taking the minimum of the sum of the fourth constraint function and the fifth constraint function as the optimization objective, optimize the poses to be utilized corresponding to the selected images to be utilized, and obtain the intermediate true pose of the robot when collecting each selected image to be utilized.

[0203] The optimization process in this step can be similar to the above steps C1 - C2. The difference is that in this step, only the true poses corresponding to a part of the images to be utilized and the true coordinates of the real positions represented by the valid feature points in this part of the images to be utilized are optimized each time, rather than optimizing the true poses corresponding to all the images to be utilized and the true coordinates of the real positions represented by the valid feature points in all the images to be utilized simultaneously. Specifically, the fourth constraint function can be expressed as: ; where represents the pose difference between the true poses of the robot when collecting the image to be utilized i1 and the image to be utilized j1, that is ; represents the true pose of the robot when collecting the currently selected image to be utilized i1; represents the true pose of the robot when collecting the currently selected image j; 、 are optimization variables. represents the pose difference between the poses to be utilized corresponding to the image to be utilized i1 and the image to be utilized j1.

[0204] The fifth constraint function can be expressed as: ; where K is the camera internal parameter of the image acquisition device, and are optimization variables, represents the true pose of the robot when collecting the currently selected image to be utilized i1, represents the coordinates of the real position represented by the valid feature point k1 in the currently selected image to be utilized i1; represents the pixel coordinates of the projection in the image to be utilized i1 of the true coordinates of the real position represented by the valid feature point k1; represents the pixel coordinates of the valid feature point k1 in the image to be utilized i1.

[0205] The objective function constructed based on the fourth constraint function and the fifth constraint function can be expressed as: 。

[0206] S1042. Using the change relationship between the intermediate true poses corresponding to each image to be utilized and the loop observation relative poses between the images to be utilized with loop relationship as constraint conditions, optimize the data of the intermediate true poses corresponding to each image to be utilized to obtain the final true poses of the robot when collecting each image to be utilized.

[0207] After obtaining the intermediate true poses corresponding to each image to be utilized in the previous step, in this step, the true pose of the robot when collecting each image to be utilized can be used as the optimization variable, and a constraint function can be constructed using the change relationship between the intermediate true poses corresponding to each image to be utilized. Furthermore, based on the constraint function constructed using the change relationship between the intermediate true poses corresponding to each image to be utilized and the constraint function constructed using the loop observation relative poses between the images to be utilized with loop relationship, optimize the data of the intermediate true poses corresponding to each image to be utilized to obtain the final true poses of the robot when collecting each image to be utilized. Specifically, this process can include the following steps:

[0208] Step D1. Construct the third constraint function and the sixth constraint function. Among them, the third constraint function represents the difference between the pose difference between the true poses of the robot when collecting every two images to be utilized with loop relationship and the loop pose difference. The loop pose difference represents the difference between the poses of the robot when collecting these two images to be utilized, determined based on the feature points with feature matching in these two images to be utilized. The sixth constraint function represents the difference between the pose difference between the true poses of the robot when collecting every two images to be utilized and the pose difference between the intermediate true poses of the robot when collecting these two images to be utilized.

[0209] The third function in this step can have the same form as the third function in the above embodiment, and will not be elaborated here.

[0210] The sixth constraint function can be expressed as: ; where represents the pose difference between the true poses of the robot when collecting image to be utilized i and image to be utilized j, that is ; represents the true pose of the robot when collecting image to be utilized i; represents the true pose of the robot when collecting image to be utilized j; 、 are optimization variables; represents the pose difference between the intermediate true poses of the robot when collecting image to be utilized i and image to be utilized j, that is 。

[0211] Step D2: Taking the minimum of the sum of the third constraint function and the sixth constraint function as the optimization objective, perform data optimization on the intermediate true poses corresponding to each image to be utilized, and obtain the final true poses of the robot when collecting each image to be utilized.

[0212] In this step, an objective function can be constructed based on the sum of the third constraint function and the sixth constraint function. For example, the objective function can be expressed as: . By calculating the optimal solution of the objective function and obtaining the values of the optimization variables when the objective function reaches the optimal solution, the final true poses of the robot when collecting each image to be utilized can be obtained.

[0213] S1043: Based on the final true poses of the robot when collecting each image to be utilized and the pixel coordinates of the valid feature points representing the same real position in each image to be utilized, calculate the final true coordinates of this real position.

[0214] Since the true coordinates of the real positions represented by the valid feature points in the images to be utilized are not optimized in the above step S1042, after obtaining the final true poses of the robot when collecting each image to be utilized, the final true coordinates of this real position can also be calculated based on the final true poses of the robot when collecting each image to be utilized and the pixel coordinates of the valid feature points representing the same real position in each image to be utilized. The calculation method can be a triangulation algorithm.

[0215] In this embodiment, accurate mapping data can be generated while reducing the cost of the robot. Further, this embodiment adopts a step-by-step optimization method for data optimization, which can reduce the computational amount in one optimization process and reduce the occupation of computing resources.

[0216] In an embodiment of the present application, the to-be-utilized poses corresponding to each image to be utilized can be obtained through the following steps:

[0217] Step F1: When the pose of the robot changes by a specified angle, perform motion estimation based on the images collected by the image acquisition device, determine the timestamp of the image collected when the robot changes by the specified angle as the first timestamp; and based on the pose collected by the odometer, determine the timestamp of the pose collected by the odometer when the robot changes the angle as the second timestamp;

[0218] Since there may be a certain time delay between the clock of the image acquisition device and the clock of the odometer, in this embodiment, time delay calibration can be performed first to align the clocks of the image acquisition device and the odometer. Specifically, as Figure 4As shown, before obtaining the to-be-utilized poses corresponding to the to-be-utilized images, the robot can be controlled to rotate by a certain angle so that the pose of the robot changes by a specified angle. During this period, the image acquisition device of the robot can continuously acquire images, and the odometer of the robot can continuously acquire the pose of the robot. At the same time, motion estimation is performed on the robot based on every two adjacent acquired images. For example, the optical flow method can be used for motion estimation. When it is determined based on the images that an angle change has occurred, that is, it is detected through motion estimation that the pose of the robot has changed by an angle during the process of acquiring two adjacent images, the first timestamp can be determined based on the timestamp of at least one of the two adjacent images. For example, the timestamp of the image acquired later among the two adjacent images can be used as the first timestamp.

[0219] In addition, when it is determined based on the odometer that an angle change has occurred, that is, an angle change has occurred between the poses of the robot acquired by the odometer at two adjacent moments, the second timestamp can be determined based on the timestamps of the odometer acquiring these two poses.

[0220] Step F2: Time-align the timestamps of the to-be-utilized images and the timestamps of the poses acquired by the odometer based on the time difference between the first timestamp and the second timestamp.

[0221] By calculating the time difference between the first timestamp and the second timestamp, the time delay calibration between the image acquisition device and the odometer can be obtained, that is, the time delay of the image acquisition device relative to the odometer. Then, this time difference can be used to time-align the timestamps of the to-be-utilized images and the timestamps of the poses acquired by the odometer. For example, the timestamp of each to-be-utilized image can be subtracted by this time difference to obtain the aligned timestamp of this to-be-utilized image. In this case, the timestamp of the aligned pose is still the original timestamp; or, the timestamp of each pose acquired by the odometer can be added by this time difference to obtain the aligned timestamp of the pose. In this case, the timestamps of the aligned to-be-utilized images are still the original timestamps.

[0222] Step F3: Interpolate the acquired poses according to the timestamps of the aligned to-be-utilized images and the timestamps of the aligned poses, so that the timestamps of the interpolated poses are the same as the timestamps of the aligned to-be-utilized images, and obtain the pose of the robot acquired by the odometer of the robot when each to-be-utilized image is acquired, as the to-be-utilized pose corresponding to this to-be-utilized image.

[0223] Since the acquisition frequencies of the image acquisition device and the odometer may be different. For example, the image acquisition device acquires one frame of image every 33 milliseconds, while the odometer acquires the pose every 10 milliseconds. In order to determine the pose to be utilized corresponding to the image to be utilized, the acquired poses can be interpolated according to the timestamps of the images to be utilized after alignment and the timestamps of the poses after alignment. For example, if the timestamp of an image to be utilized after alignment is 33 milliseconds, and the poses acquired by the odometer include Pose 1 and Pose 2, the timestamp of Pose 1 after alignment is 30 milliseconds, and the timestamp of Pose 2 after alignment is 40 milliseconds, then interpolation can be performed between Pose 1 and Pose 2 to obtain the pose at 33 milliseconds as the pose to be utilized corresponding to the image to be utilized, that is, the pose aligned with the image to be utilized.

[0224] In this embodiment, while generating accurate mapping data, the cost of the robot can be reduced. Further, when the pose of the robot changes by a specified angle, the timestamps at which the specified angle change occurs are respectively determined based on the image and the odometer to determine the time delay between the image acquisition device and the odometer. Then, the timestamps of the images to be utilized and the timestamps of the poses acquired by the odometer are time-aligned, and the acquired poses are interpolated so that the timestamps of the interpolated poses are the same as the timestamps of the images to be utilized after alignment. Finally, the pose to be utilized corresponding to the image to be utilized can be made more accurate, and the accuracy of the finally obtained mapping data can be further improved.

[0225] In an embodiment of the present application, the method for generating the mapping data may further include the following steps:

[0226] Step G1, for each valid feature point, calculate the average value of the feature similarities between the valid feature point and the valid feature points that match other features as the average similarity of the valid feature point;

[0227] The mapping data in this embodiment may further include the features of each valid feature point. Since there may be multiple valid feature points that match the features of each valid feature point, and the matching feature points actually represent the same real position, therefore, for multiple stable points that match, only the features of one stable feature point need to be retained. Specifically, for each valid feature point, calculate the feature similarity between the valid feature point and the valid feature points that match other features. For example, the Euclidean distance or cosine distance can be calculated. Then, for each valid feature point, calculate the average value of the obtained feature similarities as the average similarity of the valid feature point.

[0228] Step G2: For each real-world position represented by the valid feature points, the valid feature point with the minimum average similarity among the valid feature points representing the real-world position is used as the stable feature point for that real-world position, and the features of the determined stable feature points and the association relationships between the stable feature points and the images to be utilized are recorded in the current mapping data. Here, the images and stable feature points with an association relationship mean that the image contains the associated stable feature points or feature points that match the features of the associated stable feature points.

[0229] It can be understood that the images and stable feature points with an association relationship actually mean that the image field of view of the image contains the real-world position represented by the stable feature point. Therefore, the image may contain the associated stable feature points or feature points that match the features of the associated stable feature points.

[0230] In this embodiment, by recording the features of the determined stable feature points and the association relationships between the stable feature points and the images to be utilized in the current mapping data, a basis for subsequent positioning and map merging processes can be provided, and the specific process will be introduced below.

[0231] In an embodiment of the present application, the current mapping data further contains the global image features of the images to be utilized. After obtaining the current mapping data, as Figure 5 shown, the method may further include the following steps:

[0232] S501: Extract the global image features and the feature points included in the image to be located.

[0233] Among them, the image to be located may be an image collected by a robot when driving in a scene to be mapped. In this embodiment, the pose of the robot can be determined based on the image to be located. At this time, the global image features and the feature points included in the image to be located can be extracted first.

[0234] S502: Determine the image to be utilized whose global image features match the global image features of the image to be located as the matching image of the image to be located.

[0235] In this step, the feature similarity between the global image features of the image to be located and the global image features included in the current mapping data can be calculated, and the image to be utilized with a feature similarity greater than the preset similarity threshold is used as the matching image of the image to be located.

[0236] S503: Determine the stable feature points that match the feature points included in the image to be located from the stable feature points associated with the matching image of the image to be located.

[0237] After the matching image of the image to be located is determined in the above steps, feature matching of feature points can be performed on the image to be located and the matching image to obtain stable feature points that match the feature points in the image to be located. Since the current mapping data contains the association relationships between the stable feature points and the images to be utilized, the above step of determining the matching image actually filters the stable feature points included in the current mapping data by using the global image features, which can reduce the computational complexity of subsequent matching feature points.

[0238] S504. Calculate the pose of the robot when the image to be located is acquired based on the final true coordinates of the real position represented by the determined stable feature points, and use it as the positioning result of the image to be located.

[0239] Since the matching feature points represent the same real position, the pose of the robot when the image to be located is acquired can be calculated using the final true coordinates of the real position represented by the determined stable feature points. For example, the above PnP algorithm can be used for calculation.

[0240] In this embodiment, the features of each feature point and the global image features of each image can be extracted using a deep learning network model. This can make the features of the feature points and the global image features of each image have strong robustness to displacement, rotation, deformation, or illumination changes, etc., and can be applicable to complex scenarios, which is beneficial to improving the positioning accuracy. Moreover, the data volume of the features of the feature points and the global image features to be saved in the mapping data is relatively small, and the size of the generated mapping map does not exceed 15 MB / km. And the features of the feature points based on deep learning adopted have strong robustness, and the positioning accuracy can reach within 15 mm when used for positioning.

[0241] In this embodiment, while generating accurate mapping data, the cost of the robot can be reduced. Further, determine the image to be utilized whose global image features match the global image features of the image to be located as the matching image of the image to be located; determine the stable feature points that feature-match the feature points included in the image to be located from the stable feature points associated with the matching image of the image to be located; calculate the pose of the robot when the image to be located is acquired based on the final true coordinates of the real position represented by the determined stable feature points, which can achieve accurate positioning of the pose of the robot.

[0242] In an embodiment of the present application, the current mapping data further contains the global image features of the images to be utilized. After obtaining the current mapping data, as Figure 6 shown, the method may further include the following steps:

[0243] S601. Obtain historical mapping data, where the historical mapping data includes: global image features of historical collected images, features of historically determined stable feature points, and the final true coordinates of the real positions represented by the historically determined stable feature points; the association relationship between the historical collected images and the stable feature points.

[0244] The above historical mapping data can also be generated for a historical mapping scenario. For example, it can be generated by the mapping data generation method provided in the above embodiments of the present application. The mapping scenario to be built may have an overlapping part with the historical mapping scenario, so that the historical mapping data and the current mapping data can be merged.

[0245] S602. Perform feature matching between the global image features of each image to be utilized and the global image features of the historical collected images to obtain the images to be utilized and the historical collected images that match each other.

[0246] S603. For the images to be utilized and the historical collected images that match each other, perform feature matching between the stable feature points associated with the image to be utilized and the stable feature points associated with the historical collected image to obtain the matching stable feature points.

[0247] Similarly, in this embodiment, global image feature matching can also be performed first for preliminary screening, and then feature point matching can be performed to determine the matching stable feature points in the historical mapping data and the current mapping data.

[0248] S604. Use the matching stable feature points as the stable feature points representing the same real position, and merge the current mapping data and the historical mapping data.

[0249] Since the matching stable feature points represent the same real position, after determining the matching stable feature points, the current mapping data and the historical mapping data can be merged into the mapping data of the same map. For example, according to the matching stable feature points, the final true poses of the robot when collecting each image to be utilized in the current mapping data and the final true coordinates of the real positions represented by each stable feature point can be adjusted so that the final true coordinates of the real positions represented by the adjusted valid feature points are consistent with the final true coordinates of the real positions represented by the matching stable feature points in the historical mapping data, and for the features of the matching stable feature points in the historical mapping data and the current mapping data, only the features of one of the historical mapping data and the current mapping data need to be retained.

[0250] In this embodiment, accurate mapping data can be generated while reducing the cost of the robot. Further, by performing feature matching on the global image features of each image to be utilized and the global image features of the historically acquired images, the images to be utilized and the historically acquired images that match each other are obtained; for the images to be utilized and the historically acquired images that match each other, feature matching is performed on the stable feature points associated with the image to be utilized and the stable feature points associated with the historically acquired image, and the stable feature points that match each other are obtained; the stable feature points that match each other are used as the stable feature points representing the same real position, and the merging of the current mapping data and the historical mapping data is realized, which can support operations such as stitching and supplementing of multiple maps, and is beneficial to later maintenance and update.

[0251] In an embodiment of the present application, after obtaining the current mapping data, the method further includes the following steps:

[0252] Step H1, generating a mapping map based on the current mapping data and displaying it;

[0253] The mapping map can be displayed through the display screen of the electronic device. The displayed mapping map may include the poses and each real position of each image to be utilized in the current mapping data. For example, the poses and each real position of each image to be utilized can be displayed in the form of points.

[0254] Step H2, when receiving an editing instruction for the generated mapping map, performing an editing operation on the generated mapping map according to the editing instruction;

[0255] The above editing operation may include at least one of the following:

[0256] Performing overall translation and rotation on the coordinates of each pose and each real position included in the current mapping map;

[0257] Deleting at least one pose included in the current mapping map;

[0258] Deleting the coordinates of at least one real position included in the current mapping map.

[0259] For example, after the mapping map is displayed, the user can issue an editing instruction in any human-computer interaction manner to perform overall translation, rotation, etc. on the coordinates of each pose and each real position included in the current mapping map, so that the processed mapping map reaches a horizontal and vertical effect, that is, it is more in line with the human observation habit. And, the coordinates of the poses or real positions included in the current mapping map can also be deleted according to actual needs, and only the coordinates of the required poses or real positions are retained.

[0260] In this embodiment, it is possible to reduce the cost of the robot while generating accurate mapping data. Further, the mapping map generated by this embodiment supports an editing function, and the user can flexibly delete, modify, translate, and rotate the mapping map.

[0261] In one embodiment of the present application, as Figure 7 shown, the method may include the following steps:

[0262] S701, data preprocessing: Collect image and odometer data, calibrate the time delay and extrinsic parameters of both, and extract the deep learning feature descriptors of the image to obtain aligned image and odometer data.

[0263] That is, when the robot travels in the scene to be mapped, obtain each image to be utilized collected by the image acquisition device of the robot, as well as the pose to be utilized corresponding to each image to be utilized, and extract the global image features and the features of the feature points of the image to be utilized.

[0264] S702, data association: Select key image frames for feature matching, and construct 2D-3D observation constraints and inter-frame odometer pose constraints of the image.

[0265] That is, determine the effective feature points of feature matching in each image to be utilized, establish the association relationship between the effective feature points and the represented real positions, and determine the change relationship between the poses to be utilized corresponding to each image to be utilized, so as to construct a constraint function.

[0266] S703, loop detection: Detect the images (loops) in the same area of the key map frames, and add loop constraints.

[0267] That is, based on the pixel coordinates of the feature points of feature matching in every two images to be utilized with a loop relationship, calculate the pose change of the robot when collecting these two images to be utilized, and obtain the relative loop observation pose between these two images to be utilized; and construct a constraint function based on the relative loop observation pose between the images to be utilized with a loop relationship.

[0268] S704, map optimization: Segmentally optimize the map according to the reprojection constraints of the key map frames, the inter-frame odometer constraints, and the loop inter-frame pose constraints to obtain a globally consistent map.

[0269] That is, adopt the above process of segmental optimization to obtain the final true pose of the robot when collecting each image to be utilized; based on the final true pose of the robot when collecting each image to be utilized, and the pixel coordinates of the effective feature points representing the same real position in each image to be utilized, calculate the final true coordinates of this real position.

[0270] S705, Map Editing: Perform operations such as translation, rotation, and local deletion on a single piece of map, and perform editing operations such as splicing and supplementing on multiple pieces of map.

[0271] That is, according to the editing instructions, perform overall translation and rotation on the generated mapping map; merge the current mapping data with the historical mapping data; delete at least one pose included in the current mapping map; or delete the coordinates of at least one real position included in the current mapping map. This step is optional and is mainly used for post-editing of the map.

[0272] S706, Map Saving: According to different requirements of mapping and positioning, streamline and organize the map data structure to obtain a mapping map for post-editing and a positioning map for positioning use.

[0273] Streamlining and organizing the map data structure means that when merging map data, for the features of the stable feature points that match in the historical mapping data and the current mapping data, retain the features of one of the historical mapping data and the current mapping data.

[0274] In this embodiment, a data optimization method is adopted to optimize the real positions represented by the corresponding poses of the images to be utilized and the valid feature points included in the images to be utilized, which can reduce the errors of the finally determined real poses of the robot and the real positions represented by each valid feature point, improve the accuracy of the mapping data, and only two sensors, namely an image acquisition device and an odometer, need to be set in the robot of this solution to generate accurate mapping data. Therefore, through this solution, accurate mapping data can be generated while reducing the cost of the robot.

[0275] The embodiment of the present application also provides a device for generating mapping data, as Figure 8 shown. The device includes:

[0276] An information acquisition module 801, configured to acquire each image to be utilized collected by the image acquisition device of the robot and the corresponding pose to be utilized when the robot travels in the scene to be mapped; wherein, the pose to be utilized corresponding to each image to be utilized is: the pose of the robot collected by the odometer of the robot when collecting the image to be utilized;

[0277] A three-dimensional coordinate calculation module 802, configured to calculate the three-dimensional coordinates of the real positions represented by each valid feature point based on the pose to be utilized corresponding to each image to be utilized and the pixel coordinates of the valid feature points in each image to be utilized; wherein, any valid feature point has other feature points with feature matching, and the image to be utilized to which the other feature points with feature matching belong is adjacent to the image to be utilized to which the valid feature point belongs;

[0278] The loop observation relative pose determination module 803 is configured to calculate the pose change of the robot when acquiring the two to-be-utilized images based on the pixel coordinates of the feature points that are feature-matched in each two to-be-utilized images having a loop relationship and the three-dimensional coordinates of the real positions represented by the feature points of the feature match, and obtain the loop observation relative pose between the two to-be-utilized images;

[0279] The data optimization module 804 is configured to optimize the to-be-utilized poses corresponding to the to-be-utilized images and the three-dimensional coordinates of the real positions represented by the valid feature points by taking the change relationship between the to-be-utilized poses corresponding to the to-be-utilized images, the association relationship between each valid feature point and the represented real position, and the loop observation relative pose between the to-be-utilized images having a loop relationship as constraint conditions, and obtain the final true pose of the robot when acquiring each to-be-utilized image and the final true coordinates of the real positions represented by the valid feature points as the current mapping data.

[0280] Optionally, the data optimization module 804 includes:

[0281] The first function construction sub-module is configured to construct a first constraint function, a second constraint function, and a third constraint function; wherein, the first constraint function represents the difference between the pose difference between the true poses of the robot when acquiring each two to-be-utilized images and the pose difference between the to-be-utilized poses corresponding to the two to-be-utilized images; the second constraint function represents the difference between the pixel coordinates projected by the true coordinates of the real position represented by each valid feature point in the to-be-utilized image to which it belongs and the pixel coordinates of the valid feature point in the to-be-utilized image to which it belongs; the third constraint function represents the difference between the pose difference between the true poses of the robot when acquiring each two to-be-utilized images having a loop relationship and the loop pose difference; the loop pose difference represents the difference between the poses of the robot when acquiring the two to-be-utilized images determined based on the feature points that are feature-matched in the two to-be-utilized images;

[0282] The first data optimization sub-module is configured to optimize the to-be-utilized poses corresponding to the to-be-utilized images and the three-dimensional coordinates of the real positions represented by the valid feature points by taking the minimum of the sum of the first constraint function, the second constraint function, and the third constraint function as the optimization objective, and obtain the final true pose of the robot when acquiring each to-be-utilized image and the final true coordinates of the real positions represented by the valid feature points.

[0283] Optionally, the data optimization module 804 includes:

[0284] The intermediate true pose determination sub-module is used to optimize the data of the utilization poses corresponding to each image to be utilized, with the change relationship between the utilization poses corresponding to each image to be utilized and the association relationship between each effective feature point and the represented real position as constraints, so as to obtain the intermediate true pose of the robot when each image to be utilized is collected;

[0285] The final true pose determination sub-module is used to optimize the data of the intermediate true poses corresponding to each image to be utilized, with the change relationship between the intermediate true poses corresponding to each image to be utilized and the loop observation relative pose between the images to be utilized with loop relationship as constraints, so as to obtain the final true pose of the robot when each image to be utilized is collected;

[0286] The final true coordinate determination sub-module is used to calculate the final true coordinates of the real position based on the final true pose of the robot when each image to be utilized is collected and the pixel coordinates of the effective feature points representing the same real position in each image to be utilized.

[0287] Optionally, the intermediate true pose determination sub-module includes:

[0288] The image selection unit is used to select the first specified number of images to be utilized from the images to be utilized that have not been selected currently;

[0289] The intermediate true pose determination unit is used to optimize the data of the utilization poses corresponding to the selected images to be utilized, with the change relationship between the utilization poses corresponding to the selected images to be utilized and the association relationship between each effective feature point in the selected images to be utilized and the represented real position as constraints, so as to obtain the intermediate true pose of the robot when each selected image to be utilized is collected; trigger the image selection unit to execute the step of selecting the first specified number of images to be utilized from the images to be utilized that have not been selected currently until the intermediate true poses corresponding to each image to be utilized are obtained.

[0290] Optionally, the intermediate true pose determination unit includes:

[0291] The function construction sub-unit is used to construct the fourth constraint function and the fifth constraint function; wherein, the fourth constraint function represents the difference between the pose difference between the true poses of the robot when every two selected images to be utilized are collected and the pose difference between the utilization poses corresponding to the two images to be utilized; the fifth constraint function represents the difference between the pixel coordinates projected in the image to be utilized of the real coordinates of the real position represented by each effective feature point in the selected images to be utilized and the pixel coordinates of the effective feature point in the image to be utilized to which it belongs;

[0292] An intermediate true pose determination subunit, configured to optimize the data of the to-be-utilized pose corresponding to the selected to-be-utilized image with the objective of minimizing the sum of the fourth constraint function and the fifth constraint function, so as to obtain the intermediate true pose of the robot when each selected to-be-utilized image is acquired;

[0293] The final true pose determination submodule includes:

[0294] A function construction unit, configured to construct a third constraint function and a sixth constraint function; wherein, the third constraint function represents the difference between the pose difference between the true poses of the robot when every two to-be-utilized images with a loop closure relationship are acquired and the loop closure pose difference; the loop closure pose difference represents the difference between the poses of the robot when the two to-be-utilized images are acquired, determined based on the feature points with feature matches in the two to-be-utilized images; the sixth constraint function represents the difference between the pose difference between the true poses of the robot when every two to-be-utilized images are acquired and the pose difference between the intermediate true poses of the robot when the two to-be-utilized images are acquired;

[0295] A final true pose determination unit, configured to optimize the data of the intermediate true poses corresponding to the to-be-utilized images with the objective of minimizing the sum of the third constraint function and the sixth constraint function, so as to obtain the final true pose of the robot when each to-be-utilized image is acquired.

[0296] Optionally, the device further includes:

[0297] A rough screening module, configured to, before the loop closure observation relative pose determination module calculates the pose change of the robot when the two to-be-utilized images with a loop closure relationship are acquired based on the pixel coordinates of the feature points with feature matches in the two to-be-utilized images and the three-dimensional coordinates of the real positions represented by the feature points with feature matches, and obtain the loop closure observation relative pose between the two to-be-utilized images, for each to-be-utilized image, determine, from other to-be-utilized images whose acquisition times are before the acquisition time of this to-be-utilized image, an image whose to-be-utilized pose corresponding to this to-be-utilized image satisfies position loop closure as the rough screening image corresponding to this to-be-utilized image; wherein, the distance between the to-be-utilized poses corresponding to two images that satisfy position loop closure is less than a first distance threshold, and / or, the total distance traveled by the robot collected by the odometer between the acquisition times of the two images that satisfy position loop closure is greater than a second distance threshold;

[0298] A loop closure determination module, configured to determine, from the rough screening images corresponding to the to-be-utilized image, an image whose number of feature points with feature matches with this to-be-utilized image is greater than a preset number threshold as the image having a loop closure relationship with this to-be-utilized image.

[0299] Optionally, the loop closure determination module includes:

[0300] A similar image determination sub-module, configured to determine, from the coarsely screened images corresponding to the image to be utilized, an image whose global image feature similarity with the image to be utilized is greater than a preset similarity threshold as the similar image corresponding to the image to be utilized;

[0301] A loop determination sub-module, configured to determine, from the similar images corresponding to the image to be utilized, an image whose number of feature points that match the features of the image to be utilized is greater than a preset quantity threshold as the image having a loop relationship with the image to be utilized.

[0302] Optionally, the information acquisition module 801 includes:

[0303] A timestamp determination sub-module, configured to, when the posture of the robot undergoes a specified angular change, perform motion estimation based on the images acquired by the image acquisition device, determine the timestamp of the image acquired when the robot undergoes the specified angular change as the first timestamp; and determine the timestamp of the posture acquired by the odometer when the robot undergoes the angular change as the second timestamp based on the posture acquired by the odometer;

[0304] A time alignment sub-module, configured to perform time alignment on the timestamps of each image to be utilized and the timestamps of the postures acquired by the odometer based on the time difference between the first timestamp and the second timestamp;

[0305] An interpolation sub-module, configured to perform interpolation on the acquired postures according to the timestamps of each image to be utilized after alignment and the timestamps of the postures after alignment, so that the timestamps of the interpolated postures are the same as the timestamps of each image to be utilized after alignment, and obtain the posture of the robot acquired by the odometer of the robot when each image to be utilized is acquired as the posture to be utilized corresponding to the image to be utilized.

[0306] Optionally, each image to be utilized is obtained by frame extraction from the images acquired when the robot travels in the scene to be mapped; the difference between the postures to be utilized corresponding to every two adjacent images to be utilized is greater than a preset posture difference.

[0307] Optionally, the apparatus further includes:

[0308] An average similarity calculation module, configured to calculate, for each valid feature point, the average value of the feature similarities between the valid feature point and other valid feature points that match the features as the average similarity of the valid feature point;

[0309] A stable point determination module, which is used to, for each real position represented by valid feature points, take the valid feature point with the minimum average similarity among the valid feature points representing the real position as the stable feature point of the real position, and record the features of the determined stable feature points and the association relationships between the stable feature points and the images to be utilized in the current mapping data; wherein, the images and stable feature points with association relationships indicate that the images contain associated stable feature points or feature points that match the features of the associated stable feature points.

[0310] Optionally, the current mapping data further includes the global image features of the images to be utilized, and the apparatus further includes:

[0311] An image to be located processing module, which is used to, after generating the current mapping data, extract the global image features and the contained feature points of the image to be located;

[0312] A matching image determination module, which is used to determine the image to be utilized whose global image features match the global image features of the image to be located as the matching image of the image to be located;

[0313] A matching feature point determination module, which is used to determine the stable feature points that match the features of the feature points contained in the image to be located from the stable feature points associated with the matching image of the image to be located;

[0314] A pose calculation module, which is used to calculate the pose of the robot when collecting the image to be located based on the final true coordinates of the real positions represented by the determined stable feature points as the positioning result of the image to be located.

[0315] Optionally, the current mapping data further includes the global image features of the images to be utilized, and the apparatus further includes:

[0316] A historical mapping data acquisition module, which is used to acquire historical mapping data; wherein, the historical mapping data includes: the global image features of historical acquired images, the features of historically determined stable feature points, the final true coordinates of the real positions represented by the historically determined stable feature points; the association relationships between historical acquired images and stable feature points;

[0317] A global image feature matching module, which is used to perform feature matching on the global image features of each image to be utilized and the global image features of historical acquired images to obtain the matching images to be utilized and historical acquired images;

[0318] A stable feature point matching module, which is used to, for the matching images to be utilized and historical acquired images, perform feature matching on the stable feature points associated with the image to be utilized and the stable feature points associated with the historical acquired image to obtain the matching stable feature points;

[0319] A merging module, configured to use the matched stable feature points as the stable feature points representing the same real position, and merge the current mapping data and the historical mapping data.

[0320] Optionally, the apparatus further includes:

[0321] A mapping map generation module, configured to generate and display a mapping map based on the current mapping data;

[0322] An editing module, configured to, when receiving an editing instruction for the generated mapping map, perform an editing operation on the generated mapping map according to the editing instruction; the editing operation includes at least one of the following:

[0323] Performing an overall translation and rotation on the coordinates of each pose and each real position included in the current mapping map;

[0324] Deleting at least one pose included in the current mapping map;

[0325] Deleting the coordinates of at least one real position included in the current mapping map.

[0326] Optionally, the features of each feature point, and / or the global image features of each image are obtained by using a deep learning network model for extraction.

[0327] An embodiment of the present application further provides an electronic device, as Figure 9 shown, including:

[0328] A memory 901, configured to store a computer program;

[0329] A processor 902, configured to implement the method for generating mapping data as described in any one of the above when executing the program stored in the memory 901.

[0330] And the above electronic device may further include a communication bus and / or a communication interface, and the processor 902, the communication interface, and the memory 901 complete communication with each other through the communication bus. The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0331] The communication interface is used for communication between the above electronic device and other devices.

[0332] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0333] The aforementioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0334] In another embodiment provided by the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for generating the mapping data described in any one of the above are implemented.

[0335] In another embodiment provided by the present application, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute the method for generating the mapping data described in any one of the above embodiments.

[0336] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a solid-state disk (SSD), etc.

[0337] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes the element.

[0338] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, and computer-readable storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0339] The above are only the preferred embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application are all included in the protection scope of the present application.

Claims

1. A method for generating mapping data, characterized in that: The method comprises: Acquire each image to be used captured by the image acquisition device of the robot when the robot is driving in the scene to be mapped, and the pose to be used corresponding to each image to be used; wherein the pose to be used corresponding to each image to be used is: the pose of the robot captured by the odometer of the robot when the image to be used is captured; Based on the to-be-used postures corresponding to the to-be-used images and the pixel coordinates of the effective feature points in the to-be-used images, the three-dimensional coordinates of the real positions represented by the effective feature points are calculated; wherein any effective feature point has other feature points with feature matching, and the to-be-used images to which the other feature points with feature matching belong are adjacent to the to-be-used images to which the effective feature points belong; Based on the pixel coordinates of the feature points of feature matching in each of the two images to be used that have a loop relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, the posture change of the robot when the two images to be used are collected is calculated to obtain the loop observation relative posture between the two images to be used; wherein the two images to be used that have a loop relationship refer to: the images to be used that are collected when the robot passes the same place twice; Taking the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the represented real positions, and the relative postures of the loop observations between the images to be used with a loop relationship as constraint conditions, data optimization is performed on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, so as to obtain the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points as the current mapping data.

2. The method according to claim 1, characterized in that The method uses the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the real positions represented, and the relative postures of the loop-closed observations between the images to be used with the loop-closed relationship as constraint conditions, performs data optimization on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, and obtains the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points, including: Construct a first constraint function, a second constraint function and a third constraint function; wherein the first constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used and the pose difference between the poses to be used corresponding to the two images to be used; the second constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point projected in the image to be used and the pixel coordinates of the effective feature point in the image to be used; the third constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used with a loop relationship and the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points matched by the features in the two images to be used; Taking the minimization of the sum of the first constraint function, the second constraint function and the third constraint function as the optimization goal, data optimization is performed on the three-dimensional coordinates of the real position represented by each valid feature point and the posture to be used corresponding to each image to be used, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each valid feature point.

3. The method according to claim 1, characterized in that The method uses the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the real positions represented, and the relative postures of the loop-closed observations between the images to be used with the loop-closed relationship as constraint conditions, performs data optimization on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, and obtains the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points, including: Taking the change relationship between the postures to be used corresponding to the images to be used and the correlation relationship between the effective feature points and the represented real positions as constraint conditions, data optimization is performed on the postures to be used corresponding to the images to be used to obtain the intermediate real posture of the robot when each image to be used is collected; Taking the change relationship between the intermediate real postures corresponding to the images to be used and the relative postures of the loop observations between the images to be used with the loop relationship as the constraint conditions, the intermediate real postures corresponding to the images to be used are optimized to obtain the final real posture of the robot when collecting the images to be used; Based on the final real posture of the robot when each image to be used is collected, and the pixel coordinates of the effective feature points representing the same real position in each image to be used, the final real coordinates of the real position are calculated.

4. The method according to claim 3, characterized in that The method uses the change relationship between the postures to be used corresponding to the images to be used and the association relationship between the effective feature points and the represented real positions as constraint conditions to optimize the postures to be used corresponding to the images to be used, and obtains the intermediate real posture of the robot when each image to be used is collected, including: Selecting a first specified number of images to be used from the currently unselected images to be used; Taking the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraint conditions, data optimization is performed on the postures to be used corresponding to the selected images to be used to obtain the intermediate real posture of the robot when each selected image to be used is collected; Return to the step of selecting a first specified number of images to be used from the currently unselected images to be used, until the intermediate true poses corresponding to the images to be used are obtained.

5. The method according to claim 4, characterized in that The method uses the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraint conditions to optimize the postures to be used corresponding to the selected images to be used, and obtain the intermediate real posture of the robot when each selected image to be used is collected, including: Constructing a fourth constraint function and a fifth constraint function; wherein the fourth constraint function represents: the difference between the posture difference between the real posture of the robot when collecting each two selected images to be used, and the difference between the posture difference between the postures to be used corresponding to the two images to be used; the fifth constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point in the selected image to be used projected in the image to which it belongs, and the pixel coordinates of the effective feature point in the image to which it belongs; Taking the minimum sum of the fourth constraint function and the fifth constraint function as the optimization goal, data optimization is performed on the posture to be used corresponding to the selected image to be used, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected; The method uses the change relationship between the intermediate real postures corresponding to the images to be used and the loop-closed observation relative postures between the images to be used with a loop-closed relationship as constraint conditions, performs data optimization on the intermediate real postures corresponding to the images to be used, and obtains the final real posture of the robot when collecting the images to be used, including: Constructing a third constraint function and a sixth constraint function; wherein the third constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used that have a loop relationship, and the difference between the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points of feature matching in the two images to be used; the sixth constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used, and the pose difference between the intermediate real poses of the robot when collecting the two images to be used; Taking the minimization of the sum of the third constraint function and the sixth constraint function as the optimization goal, data optimization is performed on the intermediate real posture corresponding to each image to be used to obtain the final real posture of the robot when each image to be used is collected.

6. The method according to claim 1, characterized in that Before calculating the change in the posture of the robot when collecting the two images to be used based on the pixel coordinates of the feature points of feature matching in each of the two images to be used that have a loop relationship and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, and obtaining the relative posture of the loop observation between the two images to be used, the method further includes: For each image to be used, an image corresponding to the image to be used and having a posture to be used that satisfies the position loop is determined from other images to be used before the image to be used at a collection time, as a rough screening image corresponding to the image to be used; wherein the distance between the postures to be used corresponding to the two images satisfying the position loop is less than a first distance threshold, and / or the total distance moved by the robot collected by the odometer between the collection times of the two images satisfying the position loop is greater than a second distance threshold; From the coarse-screened images corresponding to the image to be used, an image having a number of feature points matching the image to be used that is greater than a preset threshold value is determined as an image having a loop relationship with the image to be used.

7. The method according to claim 6, characterized in that The step of determining, from the coarse-screened images corresponding to the image to be used, an image having a number of feature points matching the image to be used that is greater than a preset threshold value as an image having a loop relationship with the image to be used, comprises: From the coarse-screened images corresponding to the image to be used, an image whose global image feature similarity with the image to be used is greater than a preset similarity threshold is determined as a similar image corresponding to the image to be used; From the similar images corresponding to the image to be used, an image having a number of feature points matching the image to be used that is greater than a preset number threshold is determined as an image having a loop relationship with the image to be used.

8. The method according to claim 1, characterized in that Obtain the posture to be used corresponding to each image to be used by the following steps: When the posture of the robot changes by a specified angle, performing motion estimation based on the image captured by the image acquisition device, and determining a timestamp of the image captured when the robot changes by the specified angle as a first timestamp; and based on the posture collected by the odometer, determining a timestamp of the posture collected by the odometer when the angle of the robot changes, as a second timestamp; Based on the time difference between the first timestamp and the second timestamp, time aligning the timestamp of each image to be used and the timestamp of the pose collected by the odometer; According to the timestamps of the aligned images to be used and the timestamps of the aligned poses, the collected poses are interpolated so that the timestamps of the interpolated poses are the same as the timestamps of the aligned images to be used, and the pose of the robot collected by the odometer of the robot when collecting each image to be used is obtained as the pose to be used corresponding to the image to be used.

9. The method according to claim 1, characterized in that: Each image to be used is obtained by extracting frames of images collected when the robot is driving in the scene to be mapped; the difference between the postures to be used corresponding to each two adjacent images to be used is greater than the preset posture difference.

10. The method according to claim 1, characterized in that The method further comprises: For each valid feature point, calculate the average of the feature similarities between the valid feature point and other valid feature points matched with the feature, as the average similarity of the valid feature point; For each real position represented by the effective feature point, the effective feature point with the smallest average similarity among the effective feature points representing the real position is used as the stable feature point of the real position, and the characteristics of each determined stable feature point and the association relationship between each stable feature point and the image to be used are recorded in the current mapping data; wherein, the image and stable feature point with an associated relationship indicate that: the image contains the associated stable feature point, or contains feature points that match the characteristics of the associated stable feature point.

11. The method according to claim 10, characterized in that The current mapping data also includes global image features of the image to be used. After obtaining the current mapping data, the method further includes: Extracting the global image features and the contained feature points of the image to be located; Determine an image to be used whose global image features match the global image features of the image to be located as a matching image of the image to be located; Determining, from the stable feature points associated with the matching image of the image to be positioned, stable feature points that match the feature points contained in the image to be positioned; Based on the final real coordinates of the actual position represented by the determined stable feature points, the position and posture of the robot when the image to be positioned is collected is calculated as the positioning result of the image to be positioned.

12. The method according to claim 10, characterized in that The current mapping data also includes global image features of the image to be used, and the method further includes: Acquire historical mapping data; wherein the historical mapping data includes: global image features of historically collected images, features of historically determined stable feature points, final real coordinates of real positions represented by historically determined stable feature points; and association relationships between historically collected images and stable feature points; Perform feature matching on the global image features of each image to be used and the global image features of the historically collected images to obtain matching images to be used and historically collected images; For the matched image to be used and the historically collected image, feature matching is performed on the stable feature points associated with the image to be used and the stable feature points associated with the historically collected image to obtain matched stable feature points; The matched stable feature points are used as stable feature points representing the same real position, and the current mapping data and the historical mapping data are merged.

13. The method according to claim 1, characterized in that The method further comprises: Generate and display a mapping map based on the current mapping data; When receiving an editing instruction for the generated mapping map, performing an editing operation on the generated mapping map according to the editing instruction; The editing operation includes at least one of the following: The coordinates of each pose and each real position contained in the current map are translated and rotated as a whole; Delete at least one pose contained in the current map; Delete the coordinates of at least one real location contained in the current mapping map.

14. The method according to claim 11 or 12, characterized in that: The features of each feature point and / or the global image features of each image are extracted using a deep learning network model.

15. A device for generating mapping data, characterized in that: The device comprises: An information acquisition module is used to acquire each image to be used captured by the image acquisition device of the robot when the robot is driving in the scene to be mapped, and the posture to be used corresponding to each image to be used; wherein the posture to be used corresponding to each image to be used is: the posture of the robot captured by the odometer of the robot when the image to be used is captured; A three-dimensional coordinate calculation module is used to calculate the three-dimensional coordinates of the real position represented by each effective feature point based on the posture to be used corresponding to each image to be used and the pixel coordinates of the effective feature points in each image to be used; wherein any effective feature point has other feature points with feature matching, and the image to be used to which the other feature points with feature matching belong is adjacent to the image to be used to which the effective feature point belongs; The loop closure observation relative posture determination module is used to calculate the posture change of the robot when collecting the two images to be used based on the pixel coordinates of the feature points of feature matching in each of the two images to be used that have a loop closure relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, so as to obtain the loop closure observation relative posture between the two images to be used; wherein the two images to be used that have a loop closure relationship refer to: the images to be used that are collected when the robot passes the same place twice; The data optimization module is used to optimize the data of the posture to be used corresponding to each image to be used and the three-dimensional coordinates of the real position represented by each effective feature point based on the change relationship between the posture to be used corresponding to each image to be used, the association relationship between each effective feature point and the real position represented, and the relative posture of the loop observation between the images to be used with a loop relationship as constraints, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each effective feature point as the current mapping data.

16. The device according to claim 15, characterized in that The data optimization module comprises: The first function construction submodule is used to construct a first constraint function, a second constraint function and a third constraint function; wherein the first constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used and the pose difference between the poses to be used corresponding to the two images to be used; the second constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point projected in the image to be used and the pixel coordinates of the effective feature point in the image to be used; the third constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used with a loop relationship and the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points matched by the features in the two images to be used; A first data optimization submodule is used to optimize the data of the posture to be used corresponding to each image to be used and the three-dimensional coordinates of the real position represented by each effective feature point, taking the minimum sum of the first constraint function, the second constraint function and the third constraint function as the optimization target, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each effective feature point; and / or, The data optimization module comprises: The intermediate real posture determination submodule is used to optimize the data of the posture to be used corresponding to each image to be used, taking the change relationship between the postures to be used corresponding to each image to be used and the association relationship between each valid feature point and the represented real position as constraint conditions, so as to obtain the intermediate real posture of the robot when each image to be used is collected; The final real posture determination submodule is used to optimize the data of the intermediate real postures corresponding to the images to be used, taking the change relationship between the intermediate real postures corresponding to the images to be used and the relative postures of the loop observations between the images to be used with the loop relationship as constraints, so as to obtain the final real posture of the robot when collecting the images to be used; A final real coordinate determination submodule, used to calculate the final real coordinates of the real position based on the final real posture of the robot when each image to be used is collected, and the pixel coordinates of the effective feature points representing the same real position in each image to be used; and / or, The intermediate real pose determination submodule includes: An image selection unit, used for selecting a first specified number of images to be used from the images to be used that are not currently selected; The intermediate real posture determination unit is used to optimize the data of the posture to be used corresponding to the selected images to be used, taking the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraint conditions, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected; trigger the image selection unit to execute the step of selecting a first specified number of images to be used from the images to be used that are not currently selected, until the intermediate real posture corresponding to each image to be used is obtained; and / or, The intermediate real pose determination unit includes: A function construction subunit, used to construct a fourth constraint function and a fifth constraint function; wherein the fourth constraint function represents: the difference between the posture difference between the real posture of the robot when collecting every two selected images to be used, and the difference between the posture difference between the postures to be used corresponding to the two images to be used; the fifth constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point in the selected image to be used projected in the image to which it belongs, and the pixel coordinates of the effective feature point in the image to which it belongs; The intermediate real posture determination subunit is used to optimize the posture to be used corresponding to the selected image to be used by taking the minimum sum of the fourth constraint function and the fifth constraint function as the optimization target, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected; The final real pose determination submodule includes: A function construction unit, used to construct a third constraint function and a sixth constraint function; wherein the third constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used with a loop relationship, and the difference between the loop pose differences; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on feature points of feature matching in the two images to be used; the sixth constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used, and the pose difference between the intermediate real poses of the robot when collecting the two images to be used; A final real posture determination unit is used to optimize the data of the intermediate real posture corresponding to each image to be used by taking the minimum sum of the third constraint function and the sixth constraint function as the optimization target, so as to obtain the final real posture of the robot when each image to be used is collected; And / or, the device further comprises: A coarse screening module is used to calculate the posture change of the robot when the two images to be used are collected based on the pixel coordinates of the feature points of feature matching in each two images to be used with a loop relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, and obtain the relative posture of the loop closure observation between the two images to be used. For each image to be used, from other images to be used before the image to be used at the time of collection, an image to be used corresponding to the image to be used that satisfies the position loop is determined as the coarse screening image corresponding to the image to be used; wherein the distance between the postures to be used corresponding to the two images satisfying the position loop is less than a first distance threshold, and / or the total distance moved by the robot collected by the odometer between the collection times of the two images satisfying the position loop is greater than a second distance threshold; A loop determination module is used to determine, from the coarse screened images corresponding to the image to be used, an image whose number of feature points matching the feature of the image to be used is greater than a preset number threshold, as an image having a loop relationship with the image to be used; and / or, The loop determination module includes: A similar image determination submodule is used to determine, from the coarse-screened images corresponding to the image to be used, an image whose global image feature similarity with the image to be used is greater than a preset similarity threshold, as a similar image corresponding to the image to be used; A loop determination submodule, used to determine, from similar images corresponding to the image to be used, an image whose number of feature points matching the feature of the image to be used is greater than a preset number threshold, as an image having a loop relationship with the image to be used; and / or, The information acquisition module comprises: A timestamp determination submodule is used for, when the posture of the robot changes by a specified angle, performing motion estimation based on the images captured by the image acquisition device, determining the timestamp of the images captured when the robot changes by the specified angle as a first timestamp; and determining the timestamp of the posture captured by the odometer when the robot changes by the specified angle as a second timestamp based on the posture captured by the odometer; A time alignment submodule, configured to perform time alignment on the time stamp of each image to be used and the time stamp of the pose collected by the odometer based on the time difference between the first time stamp and the second time stamp; an interpolation submodule, for interpolating the acquired posture according to the timestamps of the aligned images to be used and the timestamps of the aligned postures, so that the timestamps of the interpolated postures are the same as the timestamps of the aligned images to be used, and obtaining the posture of the robot acquired by the odometer of the robot when acquiring each image to be used as the posture to be used corresponding to the image to be used; and / or, Each image to be used is obtained by extracting frames of images collected when the robot is driving in the scene to be mapped; the difference between the postures to be used corresponding to each two adjacent images to be used is greater than the preset posture difference; and / or, The device also includes: The average similarity calculation module is used to calculate, for each valid feature point, the average value of the feature similarities between the valid feature point and other valid feature points matched with the feature point, as the average similarity of the valid feature point; The stable point determination module is used to, for each real position represented by the effective feature point, use the effective feature point with the smallest average similarity among the effective feature points representing the real position as the stable feature point of the real position, and record the features of each determined stable feature point and the association relationship between each stable feature point and the image to be used in the current mapping data; wherein, the image and stable feature point with an association relationship means: the image contains the associated stable feature point, or contains feature points that match the features of the associated stable feature point; and / or, The current mapping data also includes global image features of the image to be used, and the device also includes: The image processing module to be located is used to extract the global image features and the feature points contained in the image to be located after generating the current mapping data; A matching image determination module, used to determine an image to be used whose global image features match the global image features of the image to be located, as a matching image of the image to be located; A matching feature point determination module, used to determine, from the stable feature points associated with the matching image of the image to be positioned, stable feature points that match the feature points contained in the image to be positioned; A posture calculation module, used to calculate the posture of the robot when collecting the image to be positioned based on the final real coordinates of the actual position represented by the determined stable feature points, as the positioning result of the image to be positioned; and / or, The current mapping data also includes global image features of the image to be used, and the device also includes: A historical mapping data acquisition module, used to acquire historical mapping data; wherein the historical mapping data includes: global image features of historically collected images, features of historically determined stable feature points, final real coordinates of real positions represented by historically determined stable feature points; and associations between historically collected images and stable feature points; A global image feature matching module is used to perform feature matching on the global image features of each image to be used and the global image features of the historically collected images to obtain matching images to be used and historically collected images; A stable feature point matching module is used to perform feature matching on the stable feature points associated with the image to be used and the historically collected image to obtain matched stable feature points. A merging module is used to use the matched stable feature points as stable feature points representing the same real position, and merge the current mapping data with the historical mapping data; and / or, The device also includes: A mapping generation module is used to generate and display a mapping map based on the current mapping data; The editing module is configured to, when receiving an editing instruction for the generated mapping map, perform an editing operation on the generated mapping map according to the editing instruction; the editing operation includes at least one of the following: The coordinates of each pose and each real position contained in the current map are translated and rotated as a whole; Delete at least one pose contained in the current map; Deleting the coordinates of at least one real location contained in the current mapping map; and / or, The features of each feature point and / or the global image features of each image are extracted using a deep learning network model.

17. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-14 when executing a program stored in a memory.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

19. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 14.

Images